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Dear Mr. Brown: 


As you requested, we examined certain aspects of the 1987-88 Nationwide Food 
Consumption Survey, a decennial survey conducted by the Human Nutrition Information 
Service, U.S. Department of Agriculture. This report makes several recommendations to the 
Secretary of Agriculture to correct data quality problems in the 1987-88 survey and improve 
methodology and contracting procedures in future surveys. 


As arranged with your office, unless you publicly announce its contents earlier, we will make 
no further distribution of this report until 10 days from the date of this letter. At that time, 
we will send copies of this report to the appropriate House and Senate committees; interested 
Members of Congress; the Secretary of Agriculture; the Administrator, Human Nutrition 
Information Service; the Administrator, Food and Nutrition Service; and the Director, Office 
of Management and Budget. We will make copies available to others upon request. 


This work was performed under the direction of John W. Harman, Director, Food and 
Agriculture Issues, who may be reached on (202) 275-5138 if you or your staff have any 
questions. Other major contributors are listed in appendix IV. 


Sincerely yours, 


’ 


J. Dexter Peach 
Assistant Comptroller General 


Executive Summary 


Purpose 


Background 


Results in Brief 


Concerns about food safety and the nutritional status of the U.S. popu- 
lation point to the need for reliable, timely information on food use and 
the dietary habits of Americans. The Department of Agriculture’s (USDA) 
Nationwide Food Consumption Survey, conducted most recently in 
1987-88, is considered a major government survey on food and nutrient 
consumption. As requested by Representative George E. Brown, Jr., GAO 
examined (1) certain aspects of the methodological soundness of the 
1987-88 survey and (2) the effectiveness of USDA’s management of the 
contractor hired to conduct the survey. 


The Nationwide Food Consumption Survey, which is conducted once 
each decade, addresses two aspects of food consumption: household 
food use, measured over 7 days, and individual food use (intake), mea- 
sured over 3 days. The survey relies on three instruments for collecting 
data—one for household and two for individual consumption. These 
instruments combine structured in-person interviews with a self-admin- 
istered questionnaire. According to a 1984 study by the National 
Research Council, the survey is widely used. The Food and Nutrition 
Service (FNS), for example, uses the data to evaluate multibillion-dollar 
federal food assistance programs. 


The purposes of the 1987-88 survey included evaluating the nutritional 
content of household and individual diets and detecting shifts in food 
use since the 1977-78 survey. The survey, conducted in the continental 
United States, comprised a “‘basic’”’ sample, which was to cover at least 
6,000 households of all incomes, and a low-income sample, which was to 
cover at least 3,600 households. Both samples were meant to be repre- 
sentative of the U.S. populations from which they were drawn. 


The contract for the survey was awarded in September 1986 for $6.2 
million, with completion expected in March 1989. uspA’s Human Nutri- 
tion Information Service (HNIS) and FNS shared responsibility for over- 
seeing the contractor, National Analysts. The HNIS contracting officer’s 
representative was responsible for monitoring National Analysts’ per- 
formance and notifying FNS’ contracting officer about any problems. 
Authority to change the contract’s work, costs, or completion dates and 
to enforce the contract’s provisions rested with FNS’ contracting officer. 


Methodological problems, deviations from the survey’s original design, 
and lax controls over the collection and processing of the results all raise 
doubts about the quality and usefulness of the data in the 1987-88 
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Principal Findings 


Poor Survey Methodology 
Compromised Data’s 
Quality 


Executive Summary 


Nationwide Food Consumption Survey. Most importantly, results from 
the survey may not be representative of the U.S. population because of 
low response rates. During its review, GAO had raised concerns that the 
data might be biased. In April 1991, an expert panel convened by HNIS to 
assess the integrity of the 1987-88 data concluded that the data may be 
biased estimates of the nation’s dietary intake. 


HNIS and FNS poorly managed the contract for 1987-88 survey, at times 
violating key internal controls designed to safeguard the government’s 
best interests. The contracting officer’s representative improperly 
approved changes without consulting the contracting officer. The con- 
tracting officer exercised no oversight during much of this time. As a 
result, the contractor did not complete key procedures required by the 
contract. These actions contributed to increasing the contract’s costs and 
delaying the contract’s completion by 2 years. 


The most serious data quality problem in the 1987-88 Nationwide Food 
Consumption Survey resulted from the low response rate for the basic 
sample: Only 34 percent of the households in the basic sample provided 
individual intake data—a response rate so low that it is questionable 
whether the data are representative of the U.S. population. The survey’s 
design may have contributed to this problem; a complex and lengthy set 
of questions was used that posed a burden for respondents. For the 
average household, the interview alone took about 3 hours. Despite the 
time investment the survey required, household members were paid 
only $2 to participate. 


In addition, the data that were collected for the 1987-88 survey may not 
be accurate because of quality control problems. The survey’s design 
required that equal numbers of households be interviewed over the four 
seasons to correct for seasonal differences in eating patterns. National 
Analysts deviated considerably from this design. Quality control 
problems also resulted from frequent staff turnover. For some new 
staff, National Analysts provided fewer hours of training than required 
by the contract. The requirement was meant to ensure that all of 
National Analysts’ staff consistently and correctly collected and 
processed the data. 
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1987-88 Survey Contract 
Was Poorly Managed 


Executive Summary 


These problems raise doubts about the integrity of the data in the 1987- 
88 survey. GAO wrote HNIS about these problems during this review. HNIS 
convened an independent panel to investigate whether the results were 
biased. The panel concluded that the data may be biased estimates of 
the nation’s dietary intake. Consequently, it did ‘“‘not recommend use of 
the data” unless users employed the greatest caution. However, since 
they are the only current data available on household and individual 
food consumption, GAO believes it is important that HNIS disclose the 
data’s limitations to the federal agencies and others that rely on the 
survey to make policy decisions. More importantly, the survey’s design 
flaws need to be corrected before HNIS conducts another nationwide food 
consumption survey. 


HNIS’ and FNS’ handling of the contract for the 1987-88 survey demon- 
strates how a federal contract should not be managed. The agencies’ 
mismanagement contributed to cost overruns, delays, and the con- 
tractor’s failure to complete certain contract tasks. 


In administering the contract, HNIS and FNS violated internal control pro- 
cedures designed to protect the public’s interest. In particular, the con- 
tracting officer’s representative frequently exceeded his authority by 
directing National Analysts to both forego certain requirements (e.g., a 
dress rehearsal to test the survey’s operations) and undertake work not 
specified (e.g., 5 months of additional data collection). He did not ade- 
quately monitor the contract or relay problems to the contracting 
officer. After GAO disclosed these management problems, the HNIS admin- 
istrator relieved the representative of his responsibilities. 


The contracting officer failed to monitor National Analysts’ contract for 
an extended period of time. Besides not communicating with the con- 
tracting officer’s representative, the contracting officer did not follow 
up on problems discussed in National Analysts’ monthly progress 
reports or on invoices showing that the contractor had spent 75 percent 
of the budget almost 1 year before the contract was to end. FNS officials 
currently involved with the contract cited several reasons why the con- 
tract was not closely monitored, including a heavy work load, their 
predecessor’s different management style, and the staff’s inexperience. 


Since 1988, the contracting officer has tried to improve the contract’s 
administration. Although FNS concedes that these efforts were too late, it 
did not terminate the contract for default after 1988 because that would 
have delayed the congressionally mandated survey. 
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Executive Summary 


GAO recommends, among other things, that the Secretary of Agriculture, 
before requesting funds for another nationwide food consumption 
survey, submit to the Congress a report (1) demonstrating that efficient 
survey instruments and procedures have been developed to reduce the 
burden on respondents, to increase respondents’ motivation to partici- 
pate in the survey, and to meet essential data needs; (2) describing a 
plan to ensure that the results obtained from the household food use and 
individual intake data are representative of the U.S. population; and (8) 
stating the steps to be taken and the quality controls to be followed so 
that future surveys will not repeat the mistakes of the past. 


USDA recognizes the reality of many of the problems GAO identified and is 
taking action to correct them. USDA also agrees with GAO’s recommenda- 
tions. Since USDA’s actions will take time to develop and implement, their 
effectiveness must be assessed in the future. These actions should be 
described in USDA’s report to the Congress. 


National Analysts commented that the data from the 1987-88 survey are 
now under review and meaningful information will be available to gov- 
ernment agencies and other users of these data. National Analysts 
wrote, “Clearly, there were problems using a new data collection method 
in as demanding and complex a survey as the Nationwide Food Con- 
sumption Survey. However, National Analysts attempted to address 
problems as they arose and performed the requirements of its contract.” 
Contrary to National Analysts’ views, the expert panel convened by HNIS 
confirmed GAO’s findings that the survey has serious limitations largely 
due to the low response rates. Furthermore, as GAO documents in this 
report, National Analysts did not perform all the contract’s 
requirements. 


USDA and National Analysts commented on other more technical aspects 
of this report. GAO revised the report where appropriate. 
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Chapter 1 


Introduction 


Several government agencies collect and analyze data to measure food 
and nutrient consumption and the health and nutrition status of the U.S. 
population. The U.S. Department of Agriculture’s (USDA) Nationwide 
Food Consumption Survey (NFCS), conducted most recently in 1987-88, is 
considered a leading government survey in this area.' It addresses two 
aspects of food consumption—household food use and individual food 
intake. USDA’s Human Nutrition Information Service (HNIS) planned and 
supervised the 1987-88 survey, which was conducted by a contractor, 
National Analysts.’ 


According to a 1984 National Research Council (NRC) study, NFcs data 
are important to a multiplicity of users in government, the academic 
community, and industry.? Moreover, according to the study, the survey 
serves a distinct and important purpose that no other survey currently 
produced can fulfill. Data from the NFCS answer some basic questions 
not covered by other surveys, detailing, for example, who in the popula- 
tion consumes what foods. This information is needed not only to assess 
the nutritional status of the nation but also to design, analyze, or modify 
a variety of food assistance, education, and regulatory programs. HNIS 
and another USDA agency, the Food and Nutrition Service (FNS), rely on 
the data for food assistance programs, an area that received about $24 
billion in appropriations in 1991. HNIS uses the data on low-income 
households to update the Thrifty Food Plan.‘ FNS uses NFCS data to deter- 
mine the effectiveness of food assistance programs, to target services, 
and to estimate the effects of legislative and economic changes. The 
survey also provides data for other federal activities, such as the Envi- 
ronmental Protection Agency’s (EPA) regulation of human exposure to 
pesticide residues on foods.® Because the data are important to many 
programs, it is critical that the survey be accurate and timely. 


'The other major survey is the Department of Health and Human Service’s National Health and 
Nutrition Examination Survey. The third such survey will collect data between 1988 and 1994 from 
44,000 individuals on the prevalence of specific diseases and conditions, the dietary intake of individ- 
uals, and their health status. 


2National Analysts is a division of Booz, Allen & Hamilton, Inc. 


3National Survey Data on Food Consumption: Uses and Recommendations, National Research Council 
(July 1984). 


4USDA prepares guides for selecting nutritious diets at different levels of cost. The food plans 
describe the amounts of different food groups that households can use to provide nutritious diets for 
their families at four cost levels—-liberal, moderate, low-cost, and thrifty. The Thrifty Food Plan is 
the standard for benefit levels in the Food Stamp Program. 


5EPA registers pesticides and establishes maximum allowable pesticide residues in or on food. Key 
elements in regulating safe levels of pesticide residues are the types and quantities of food people eat. 
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Chapter 1 
Introduction 


Under its general mission to promote the food and agricultural sciences, 
USDA has conducted seven NFCSs, one about every 10 years since 1936. 
The early surveys covered household food consumption, but in 1965 
their scope was expanded to include individual food intake.° The Food 
and Agriculture Act of 1977 (P.L. 95-113) designated USDA the lead 
agency for research in food and agricultural sciences. HNIS is the agency 
responsible for conducting applied research in food and nutrition 
(except for biomedical aspects of human nutrition concerned with the 
diagnosis or treatment of disease). Most of HNIS’ annual appropriations 
directly support research, analysis, and technical assistance. 


The purposes of the 1987-88 NFCS were to evaluate the nutritional con- 
tent of household and individual diets and to detect shifts in food use 
since the previous survey. The 1987-88 NFcS comprised a “‘basic”’ 
sample, which was to cover at least 6,000 households of all income 
levels in the continental United States, and a supplemental low-income 
sample, which was to cover at least 3,600 households with incomes that 
were 130 percent of the poverty level or below. On the basis of census 
figures on average household size, the basic and low-income samples 
were expected to obtain food intake data from 16,200 and 9,720 individ- 
uals, respectively. The samples were to be representative probability 
samples, evenly distributed over the four quarters of a 12-month period 
(beginning April 1987) as well as over the days of the week so as to 
minimize differences in the data due to weekday and seasonal variations 
in food consumption. HNIS planned to have the 1987-88 NFCS contract 
conducted between September 1986 and March 1989 at a cost of $6.2 
million. The contract was completed in April 1991, and its final cost was 
$7.6 million. 


Under the terms of the contract, National Analysts, a commercial survey 
research firm in Philadelphia, Pennsylvania, was responsible for devel- 
oping designs for the basic and low-income samples; formatting the final 
survey questionnaires; collecting data from households, including those 
that did not respond to the questionnaire; and processing the data from 
returned questionnaires into final data tapes. Under contracts with HNIS 
(or its predecessor agencies), National Analysts has conducted several 
USDA food consumption surveys since 1955, including the 1965-66 NFCs; 
the 1977-78 NFCS, with supplementary surveys of elderly, low- income, 
Puerto Rican, Alaskan, and Hawaiian populations; and the two contracts 


®The most recent NFCS for which complete data are available was conducted in 1977-78. In 1979-80, 
a supplemental NFCS for low-income households was conducted. The next NFCS is expected to be 
conducted in 1997-98. 
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for the Continuing Surveys of Food Intakes by Individuals in 1985, 
1986, and 1989 through 1991. 


HNIS and FNS shared responsibility for overseeing National Analysts. HNIS 
financed the contract with appropriated funds and developed the tech- 
nical requirements for the contract, specifying all tasks and deliver- 
ables, including edited data tapes and operations reports. HNIS was 
responsible for monitoring the contract and ensuring that it received 
deliverables as specified in the contract. FNS provided administrative 
support to HNIS. As the government’s business agent, FNS’ contracting 
officer was responsible for negotiating the NFcs contract and ensuring 
that all terms and conditions were enforced. Thus, HNIS was to notify FNS 
about any real or potential problems with National Analysts’ perform- 
ance so that appropriate action could be taken to enforce or modify the 
contract and fully protect the government’s interests. 


Objectives, Scope, and 
Methodology 


In response to a request from Representative George E. Brown, Jr., we 
examined (1) certain aspects of the 1987-88 NFcS’ methodological sound- 
ness and (2) the effectiveness of USDA’s management of the contractor 
hired to conduct the survey. 


To determine the methodological soundness of the 1987-88 NFCS, we 
reviewed documents from HNIS and the contractor concerning method- 
ological research on dietary intake surveys and the development of the 
automated (computerized) method for collecting household food con- 
sumption data, the survey’s operations, preliminary and final data tapes 
from the basic sample for individual intake, and documents released to 
researchers on the data. Because National Analysts had not delivered all 
data to HNIS at the time of our review, we were able to examine only the 
data tapes used for the basic sample and the tabulations the contractor 
prepared for the low-income sample to investigate issues concerning the 
survey’s response rates. We interviewed several USDA and National Ana- 
lysts officials about the survey’s design, sampling and statistical issues, 
and matters concerning response rates and operations. We reviewed sta- 
tistical guidance for government surveys and other literature on such 
topics as automated data collection methods and sources of error in 
surveys to develop criteria or standards for evaluating the quality of the 
NFCS. Outside experts in statistical surveys reviewed our drafts.’ 


7 Joseph Steinberg, President of Survey Design, Inc. (and consultant to USDA in 1977), and Thomas 
Jabine, independent consultant. 
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To assess both the effectiveness of USDA’s management of National Ana- 
lysts and the contractor’s performance, we reviewed federal acquisition 
regulations concerning standards for performance. We reviewed docu- 
ments at FNS and HNIS, including the solicitation and modifications of the 
contract, and correspondence both with agency officials and with the 
contractor. We interviewed current and former HNIS and FNS officials 
involved with the NFCs contract regarding a wide range of activities and 
events, including the development, the solicitation, and modifications of 
the contract; communication between FNS, HNIS, and National Analysts; 
contract administration; and the contractor’s performance. We inter- 
viewed National Analysts representatives about various contract mat- 
ters and obtained documents from them regarding the performance of 
tasks and the status or outcome of deliverables, such as the follow-up 
surveys of nonrespondents. 


We conducted our review between March and November 1990 in accor- 
dance with generally accepted government auditing standards. USDA pro- 
vided formal written comments on a draft of this report (see app. I). 
These comments are presented and evaluated in chapters 2 and 3. 
National Analysts provided formal written comments on an excerpt 
from the draft report containing factual information (see app. IJ). 
National Analysts’ comments are presented and evaluated in chapters 2 
and 3. 
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Poor Methodology Compromised Quality of 
1987-88 NFCS Data 


Survey’s Design 


Two Parts of Survey 


Flaws in the methodology of the 1987-88 NFcs, deviations from the orig- 
inal design, as well as lax controls over the collection and processing of 
the survey’s results all raise doubts about the quality of the data 
obtained. Most importantly, the survey may not be representative of the 
U.S. population because it had a very poor response rate: About two- 
thirds of the households in the basic sample did not respond. In the 
absence of incentives to participate, households may have been discour- 
aged by the length and complexity of the NFCS questionnaire—a problem 
that had surfaced previously, when National Analysts collected data for 
the 1977-78 NFCS. 


HNIS convened an independent panel of experts to review the integrity of 
a component of the NFcs data. We had raised concerns about their pos- 
sible bias during our review. In its April 1991 draft report, the panel 
wrote that, because the data might be biased, it did “not recommend use 
of the 1987-88 NFcs data” unless users employed the greatest caution. 
The panel’s findings, which HNIS plans to publish, confirm and describe 
just how limited the NFcs data are. This information should enable agen- 
cies and others to decide if their need for the data and their tolerance 
for possibly substantially biased estimates outweigh the data’s 
limitations. 


The 1987-88 NFcs, largely unchanged from the design of the previous 
1977-78 survey, involved lengthy and complex questionnaires requiring 
extensive information from household members on food consumption 
and individual food intake. National Analysts began by presenting 
households with a form letter from USDA explaining the purpose of the 
survey and asking for their participation. If the household’s main meal 
preparer agreed to participate, National Analysts instructed that person 
how to prepare for the interview, completed a short series of questions 
(called a screener questionnaire), and set a time to administer the ques- 
tionnaire 7 days later. 


The survey’s household food consumption component, which was con- 
ducted first, measured the “‘disappearance’”’ of foods—that is, the 
amount of food consumed or disposed of—from home food supplies over 
a 7-day period. The household’s main meal preparer was asked to pro- 
vide information on 21 food groups (each comprising more detailed cate- 
gories) used by the household and the cost of that food (see fig. 2.1 for 
an excerpt for one food group). Questions covered the household’s com- 
position, income, and other socioeconomic characteristics, participation 
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in food assistance programs, and food expenditures and buying 
practices. 


After the household interview was completed, the individual intake 
component of the survey was administered. All household members 
(except for boarders, employees, or guests) aged 12 and older were 
asked to recall the food eaten the previous day. Following that, the 
interviewer gave each member 2-day records and explained how to fill 
in the records for foods eaten earlier on the day of the interview as well 
as the next day. The main meal preparer was asked to complete the 
form for each child under 12 years old. Records were left for absent 
members to complete after they were instructed by the main meal 
preparer. Detailed information was requested on what foods and bever- 
ages were consumed and how much, when they were consumed, and 
whether they were obtained from home or away from home. Examples 
of acceptable entries are (1) boneless chicken breast, roasted, 1 slice, 2 
inches by 1-1/2 inches by 1/4 inches, and (2) 1-1/2-cup scoop of vanilla 
ice cream with | tablespoon of fudge sauce and 1/2 teaspoon of crushed 
walnuts. The intake forms also asked for information on (1) the con- 
sumption of water, fats, salt, and vitamin and mineral supplements and 
(2) diet and health factors, including food sufficiency, frequency of 
eating certain foods, smoking, and alcohol use. When the interviewers 
returned in a few days to pick up completed forms, they were to review 
them for completeness and detail. Each participant was paid $2 for com- 
pleting the 3 days of intake; payment was not to exceed $20 per 
household. 


Other design features were intended to provide controls over or infor- 
mation about the quality of the survey’s execution. For example, the 
contract required a full-scale ‘dress rehearsal” in advance of the 
planned start of data collection in April 1987 in order to demonstrate 
National Analysts’ readiness and to uncover procedural problems that 
might need correction. All aspects of the survey were to be practiced 
during the dress rehearsal, from the interviews, using the full household 
and individual intake questionnaires, to the processing of data to pro- 
duce final data tapes. The dress rehearsal was to involve interviews 
with up to 100 households close to Philadelphia that were not part of 
the nationwide sample. 
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Figure 2.1: Excerpt From Household Interview Used in 1987-88 NFCS 


1. In the past seven days did your household use any vegetables? = 5 
2. Did you use any (FOOD CATHOORY)? READ ITEMS IN BOXES. ASK Q's 3 TO 12 BELOW IN ORDER FOR EACH "YES." ) (SKIP TO mo | 2 | 
: 
1 Globe or French 1b 37 Without tops 56 68 Root 81(5) Fresh, trimmed 
2 Jerusalem (sunchoke) 16 38 With tops 69 Commercially 82(S) Fresh, bulk, 

3 Artichoke hearts 17 


ao Gevee Mw am 
#0 [came 70(5) Fresh, trinmed 


Bayo 

Black 

Brown 

Calico 

Fava 

Kidney 

Mixed 

Pinto 

Red Mexican 
24 Refried beans 

Shellie 


59(5) Dill, fresh 71(5) Fresh, bulk, not 
41 Celeriac, knob celery trimmed 85 
lel he ne 
6 |BAKED BEANS (WITH CR 25 42 86 Spanish onions 
WITHOUT PORK CR TOMATOES) 73 Kohlrabi 87(8) Onion flakes 
26 43 Chives ENDIVE/CHICORY 74 Larbs quarters 88 hin rings 
75 Leeks 
27 Alfalfa sprouts 61 Green, curly-leaf GREEN CNICNS 
eS, aise > 
ij Not trimmed 44(5) Fresh, trinmed French 89(5) With tops, fresh 
8 Trimmed 45(S5) Fresh, bulk, mot trimmed 63 Escarole 


2% Without tops 46s ther [erme| 90(5) Without tops, fresh 
3 ite 
30 Greens aly Headed: 91 
9 Mature, dry 64 Not dry flakes 71(5) Iceberg 


(no beets) 
Yellow: 93(8) Dry flakes 78(5) Boston, bibb 

10(5) In pod, fresh 31 47 Tn husk 79(5) Leaf, Cos, 92 
u Not. in pod 48 Not. in husk, on cob 65 romaine, salad 
Qut aff cob bowl, Simpson, 

a So 

White: 

: 2 

Geen ar white 

Red cabbage 

Chinese 


12(5) Not trimmed, fresh 
13 i 


14 |WHITE GEANS (NAVY, PEA CR 
GREAT NORTHERN, MARROW) 


Quality control procedures for training and supervising survey per- 
sonnel were to be consistently followed. Standardized training of inter- 
viewers and data processing staff is important to reduce errors in 
responses to questionnaires and in the conversion of responses into 
quantitative data. Other quality control steps included adhering to data 
collection procedures to maximize response rates, documenting the 
number and results of attempts to contact a household, obtaining infor- 
mation about nonrespondents, and documenting data processing proce- 
dures for coding responses and entering them into the final data tapes. 
In addition, interviewers were to be debriefed before they terminated 
employment so that National Analysts could obtain information about 
the data collection process and solicit the interviewers’ opinions about 
the data’s quality. 
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0.4 In what fo.5 (ask [0.6 (askfo.7 In the past 7 days, altogether [0.8 Did you [0.9 (IF BOUGHT) when you baht 0.10 (IF. 
form was the {IF “COMER-|IF "COM- [how many pounds and ounces did you [buy the (FOOD [the (FOOD ITEM) you used, how many(BOUGHT) j(IF 
(FOD TIE) CIA ERCIA ise? (PROBE: How much did you use?) |TTEM), home pounds and aunces did you buy? at did JBOUGHT) 
you FROZEN OR CANNED” [PROBE FOR # AND TYPE OF UNITS produce it, orjPROBE FOR # AND TYPE OF UNITS ou pay {Is 
rephlacreds Syrrg: IN NGS) receive it as for the jthat 
a gift or as (TOTAL {the 
Da = ? AMO total 
IN price 
0.9)? fyou 
saxe? 


TORN 


ia Ha 
a (EAE ONS 
Cer EE SS ee ee 
Cer ae SS ee 
LRP pe a de Ls Le LET | loeb! [TTP folobalslad | dad 
te eee ee iogcecee Eel 
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Method ology of 1987- Questions about the NFcs’ methodology were raised before the survey 


began in April 1987. In a report on the 1977-78 NFCS, we criticized the 
88 NFCS Was Flawed use of a long questionnaire that seemed to place an unnecessary burden 
on respondents and may have jeopardized the quality of their 
responses.! A subsequent study for USDA also raised concerns about the 
potential for a low response rate, as well as other methodological 
points.? HNIS addressed many of these issues in research projects during 
the 1980s, but for several reasons left the survey’s design basically 
unchanged. Rather than correcting the problems, HNIS may have added 


1Nationwide Food Consumption Survey: Need for Improvement and Expansion (CED-77-56, Mar. 25, 
1977). 


2Recommended Survey Design for Validation of Findings of Nationwide Food Consumption Survey, 
1977-78, Survey Design, Inc. (Oct. 1977). 
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new problems by introducing a poorly developed computerized method 
of collecting the data. 


In 1977, we raised concerns about the methodology planned for the 
1977-78 NFcs. Among them, the household questionnaire was long, 
placing an unnecessary burden on respondents that might harm the 
quality of their responses. We questioned the “‘list-recall” method of 
exposing each respondent to a list of a large number of choices in each 
main food group to aid their recollection when only a small portion of 
the choices would be relevant. We also reported that for some partici- 
pants who used all main food groups, having to listen to the reading of 
up to 350 food categories and then wait for the interviewer to find the 
code from more detailed subgroups was an unnecessary burden.? USDA 
had not adequately reviewed alternatives, including reducing the 
number of food subgroups to those with nutritional differences and/or 
having the interviewer code the data later. 


USDA responded in 1977 that we had overestimated the burden of the 
list-recall method and that the alternatives would be more burdensome 
and would reduce the validity of the responses. However, USDA asked an 
outside consultant to recommend ways to improve the quality of future 
studies. The consultant recommended several research efforts to 
improve the methodologies for the survey’s household and individual 
intake components. The consultant also raised concerns about reducing 
the potential for a low response rate. As it turned out, response rates for 
the 1977-78 survey were low. The contractor’s 1980 report showed that 
only 61 percent of the targeted households participated in the survey! 
(the net individual response rate was 57 percent). 


HNIS completed many of the methodological studies the consultant rec- 
ommended for the survey, though some remain on the research agenda 
for the future.® However, according to HNIS, less methodological research 
has been done for the household food consumption component or the 


3The 1987-88 household questionnaire increased the number of food categories that potentially had 
to be read to the respondent to 398. 


4Of 24,408 occupied units, 14,964 households completed interviews. 
5HNIS has summarized the results of these studies and laid out its agenda for future research in 


USDA Methodological Research for Large-Scale Dietary Intake Surveys, 1975-88 (Home Economics 
Research Report No. 49, Dec. 1989). 
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NFCS as a whole in the last two decades because, in preparation for intro- 


Computerized Method Was 
Poorly Developed 


ducing new surveys, HNIS has been investigating methodologies for col- 
lecting individual intake data. HNIS changed its approach to food 
consumption surveys after the 1977-78 NFcs. In 1985, HNIS began yearly 
surveys of individual diets in order to obtain continuous estimates of the 
dietary status of various sex/age groups. 


Recognizing that the burden on respondents would be reduced if the 
household and individual components were separated, HNIS considered 
this option for the 1987-88 NFcs. However, HNIS rejected separating the 
components after deciding that the need for information that could be 
derived only from linked data sets was critical. HNIS considered linked 
data critical to developing allowances for the Thrifty Food Plan and 
food stamps. 


HNIS compounded the flaws in the NFCS’ design by prematurely intro- 
ducing a major methodological change from previous surveys: use of 
computer-assisted personal interviewing (CAPI), in which interviewers 
asked the questions from the household questionnaire and entered 
responses directly into a laptop computer. This method was primarily 
introduced to shorten the time needed for data processing. According to 
National Analysts and HNIS officials, many interviewers were not skilled 
in using the computer and subsequently quit because they were dissatis- 
fied with the CAPI method. 


These problems might have been avoided if HNIS and National Analysts 
had better developed and fully tested the CAPI method before implemen- 
tation. In 1986, another HNIS contractor, Creative Associates, Inc., 
pretested an automated version of the household questionnaire on 63 
households, using experienced interviewers and cooperative respon- 
dents. (This pretest was a scaled-down version of a much larger field 
test proposed by HNIS in 1985 but rejected by the Office of Management 
and Budget (OMB) because of its cost.) Results from the pretest alerted 
HNIS to the need for major revisions, some of which were made for the 
NFCS. Both interviewers and respondents complained about fatigue 
because the interview took so long—from 1-1/3 to 5-1/2 hours. Some 
respondents reportedly refused to finish their interviews because of the 
length. In addition, the accuracy of both the reported and the recorded 
data declined as the interview progressed. Interviewers reported anx- 
iety and frustration at not being adequately trained to use the computer 
and not having enough practice. 
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The pretest suggested that if the interviewers’ concerns were ignored, 
the turnover in interviewers would be high and the quality of the data 
low. Nevertheless, HNIS officials believed that the improvements made 
by National Analysts would address the interviewers’ problems. The 
NFCS contract allowed National Analysts to develop new CAPI procedures 
using technically superior software and computer equipment. However, 
National Analysts’ development of the CAPI method was not successful in 
overcoming the interviewers’ apparent frustrations with the method. 
According to a former HNIS official, if major problems were found early 
on, National Analysts could resort to the traditional “paper and pencil” 
questionnaire for the entire survey. The HNIS official who oversaw the 
contractor, however, stated that the contract would have had to be mod- 
ified if a paper and pencil version were used. 


The problems with the cAPI method also might have been averted if 
National Analysts had had enough time to adequately develop and test 
its CAPI version of the NFCS questionnaire before data collection began in 
April 1987. The contractor pretested only two abbreviated versions— 
though the results of both revealed the same problems that had surfaced 
earlier, problems that could be overcome only if interviewers were rigor- 
ously trained. The scheduled dress rehearsal would have afforded yet 
another opportunity to discover problems with the cAPI method, but HNIS 
and National Analysts canceled the dress rehearsal to save time after 
the contractor had fallen behind schedule. 


Two major circumstances during the first months of the 1987-88 NFcs 
led to extensive deficiencies in collecting data and processing the results. 
First, two-thirds of the targeted households did not provide any indi- 
vidual intake data. Second, many of the contractor’s trained inter- 
viewers and data processing staff resigned. As a result, the contractor 
deviated from procedures specified in the survey’s design in ways that 
may have undermined the data’s quality. 


One of HNIS’ priorities for the 1987-88 NFCS was that the response rates 
be improved over those of earlier surveys. Under the contract, National 
Analysts aimed for a 74-percent response rate from households and an 
85-percent response rate from individuals within the cooperating house- 
holds. However, the contractor collected data from far fewer households 
and their members than originally planned. 
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Response rates for the NFCS’ individual intake component for the basic 
sample were poor, whether measured by the number of households or 
the number of individuals that responded.* Only 4,114 of 13,733 
targeted households provided any individual intake information, or 33.8 
percent, excluding vacant units. Thus, for every 100 households 
targeted, almost 70 had members who did not provide any intake infor- 
mation.’ An estimated 31.1 percent of the eligible individuals in the 
sample provided food intake information for at least 1 day, and 25.9 
percent for the full 3 days. (See app. III for details on how we calculated 
response rates. ) 


Several factors may explain why participation in the survey was low. 
Steps that might have encouraged respondents to cooperate were not 
taken. For example, although HNIS’ methodological research showed that 
response rates would improve if sample households were given advance 
notice of the survey, the contract allowed National Analysts’ inter- 
viewers to make unannounced visits so that they could more easily 
arrange their work schedules. Moreover, incentives to participate, such 
as payment more commensurate with the time required, were lacking. 
According to HNIS officials, HNIS has been unable to obtain approval from 
OMB to increase the $2 payment for participants. 


In the absence of compelling incentives, however, the greatest factor dis- 
couraging participation may have been the burden the survey placed on 
respondents. According to one of National Analysts’ progress reports, 
submitted to HNIS during the survey, many households refused to par- 
ticipate solely because of the time required. In March 1988, National 
Analysts reported to HNIS that the single most significant deterrent to 
cooperation was that households had to be told beforehand that the 
survey would involve a lengthy interview. For the average household of 
about three people, completing the household interview and the first 
day of intake records required almost 3 hours (no data are available on 
the time required to complete the remaining 2 days of intake records). 
HNIS has reported that the time involved in conducting the household 


SBecause many data were still pending from National Analysts at the time of our review, we were 
able to obtain complete information only for the individual intake component for the basic sample. 
We were unable to determine final response rates for the 7-day household food use component for the 
basic sample or either component of the survey for the low-income sample. However, our analysis of 
preliminary tabulations from National Analysts showed that response rates were between 33 and 45 
percent. 


7We do not know how many individuals were in the 13,733 households in the sample. To estimate 


response rates for individuals who provided 1 or 3 days of intake data, we assumed that the average 
household sizes of both households that participated and those that did not were the same. 
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and individual components of the 1987-88 NFcs together placed an 
excessive burden on respondents, which contributed to the low response 
rates. As a result, HNIS is again considering establishing the individual 
intake component and the household component as separate surveys. 
However, HNIS noted that this might affect USDA’s development of the 
Thrifty Food Plan, which requires data from both components. 


The NFCS was designed to evaluate the nutritional content of U.S. house- 
hold and individual diets on the basis of an unbiased estimate, that is, a 
sample that, over repeated surveys, would have the same expected 
values as the population from which it was drawn. Because National 
Analysts failed to collect data from many of those selected to be in the 
sample, however, the possibility exists that the data may not be repre- 
sentative of the U.S. population because those households that did not 
respond (nonrespondents) may have differed significantly from those 
that did with regard to the characteristic measured—food consumption. 


National Analysts was to develop a profile of nonrespondents through a 
mail-out questionnaire and interviews with subsamples of (1) house- 
holds that would not participate in a complete interview, (2) their neigh- 
bors, or (8) other proxy sources. The aim of this profile—a standard 
technique—was to determine whether nonrespondents were systemati- 
cally different from respondents. 


However, in March 1991, National Analysts informed HNIS that the data 
on nonrespondents were unavailable. According to National Analysts’ 
officer-in-charge, the questionnaire and the documents on the follow-up 
interviews with nonrespondents were lost during an office move in Jan- 
uary 1990, and the data were never entered into a data base. Although 
she told us that they mailed the nonrespondent questionnaire and con- 
ducted interviews sometime in the fall of 1988, National Analysts gave 
us no documents to substantiate that these tasks were done. In 
explaining the loss and inattention to the processing of records resulting 
from these efforts, the officers-in-charge said that the surveys of 
nonrespondents were the ‘“‘orphans”’ of the project. 


National Analysts did not adequately implement another key feature of 
the survey’s design: distribution of the household interviews across the 
four seasons (quarters) of the year to minimize variations in data due to 
seasonal differences in food consumption. By the end of the first quarter 
in June 1987, only 638 of the 2,400 interviews expected were completed. 
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Consequently, HNIS and National Analysts jointly decided to extend the 
first quarter’s data collection into the next quarter (summer 1987). Ulti- 
mately, National Analysts’ interviewers were simultaneously collecting 
data across all quarters, regardless of when the household was sched- 
uled to be interviewed under the original design. Rather than adhere to 
the 1-year data collection plan, National Analysts extended interviewing 
by 5 months, further disrupting the quarterly design. (See table 2.1.) 


HNIS has since adjusted the data by weighting the results differently for 
each day and month. For example, results for Sunday interviews, which 
accounted for only 5 percent of the sample, were adjusted upward so 
that they represented one-seventh, or 14 percent, of the interviews. 


HNIS has also reweighted the data using 13 demographic characteristics 
from the Census Bureau’s March 1987 Current Population Survey. 
Under this weighting scheme, each individual received a weight 
reflecting that individual’s contribution relative to the 1987 U.S. popula- 
tion, with the sum of the weights equal to the total population. At least 
one individual represented as few as 1,000 persons and as many as 
136,000 persons. This large disparity decreased the precision of esti- 
mates that could be derived from the 1987-88 NFcs.8 


Tc ae ie a se ES ES 8 Ae ee a 
Percentage of 


Season? interviews conducted 
Spring (1987 and 1988) 4 
Summer (1987 and 1988) 15 
Fall (1987) 16 
Winter (1988) 28 


*The months comprising each season were April-June (spring), July-September (summer), October- 
December (fall), January-March (winter). 


8Since the NFCS started with a probability sample to develop estimates, each estimate has a measure- 
able precision, or sampling error, which is expressed as a plus/minus figure with a specified degree of 
confidence. A sampling error indicates how closely we can reproduce from a sample the results that 
we would obtain if we were to take a complete count of the universe using the same measurement 
methods. By adding the sampling error to and subtracting it from the estimate, we can develop upper 
and lower bounds for each estimate. This range is called a confidence interval. Sampling errors and 
confidence intervals are stated at a certain confidence level. For example, a confidence interval at the 
95-percent confidence level means that in 95 out of 100 instances, the sampling procedure used 
should produce a confidence interval containing the universe value being estimated. 
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National Analysts introduced other design changes after the first 
quarter, in June 1987, when interviewers had resigned unexpectedly, 
forcing the company to hire and train new employees.’ These changes 
may have jeopardized the quality of the data. According to USDA officials 
who observed the training of interviewers early in the survey, training 
on the computer method was inadequate. Both National Analysts and 
HNIS reported that interviewers had many difficulties using the com- 
puter method. These difficulties could have been avoided with better 
training. 


On a related matter, two months into the survey, the training period was 
shortened to less than 5 days for newly hired interviewers. This fact 
raises the possibility that new employees were inadequately trained to 
handle problems or follow standard procedures. Some interviewers 
began working before being trained on the computer method, and not all 
interviewers used the computer. Consequently, household interviews 
were administered differently—most by computer, but some with a 
printed version of the questionnaire. Three of the five NFCS interviewers 
we contacted on a nonrandom basis said that they never used the com- 
puter to conduct the interview, and one reported having had to admin- 
ister the printed version without being trained to do so. Differences in 
administration can change how a question is asked—for example, inter- 
viewers may probe differently with a paper questionnaire than with a 
computerized questionnaire—and can introduce unintended variation in 
answers. National Analysts also did not document its debriefings of 
interviewers who quit, as required by the contract. This left gaps in 
information about how their application of the computer method, for 
example, might have affected the quality of the data collected. 


Quality controls were also relaxed for the data processing staff, particu- 
larly coders.'° This may have jeopardized the quality of the NFCS data. 
According to HNIS officials, the data processing was poor in quality and 
was unreasonably delayed. As required by its contract, National Ana- 
lysts had developed detailed procedures for ‘‘post-field” processing of 
NFCS data. Among these procedures, job applicants were to be screened 
for certain skills, newly hired reviewers and coders were to receive 5 
days of intensive training, and supervisors were to systematically 


Of the 240 interviewers who collected individual intake data for the basic sample, 40 percent worked 
for 3 months or less on the survey. Only 21 percent worked on the survey for more than 1 year. 


10Transforming qualitative information collected in a survey, like descriptions of foods eaten, into 


quantitative data for subsequent analysis is called data coding and reduction. Trained coders trans- 
late a survey’s qualitative data into quantitative data using established codes and rules. 
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review and verify the work of each coder to ensure that they applied 
codes consistently. These procedures were not followed, though details 
about their implementation are obscure because, according to the con- 
tractor, no documentation exists. When a large number of trained coders 
reportedly resigned or were released because of poor performance, 
National Analysts subcontracted with temporary employment agencies 
to fill these positions. The company could not provide us with any 
details on the number of the temporary coders who worked on the 1987- 
88 NFCS or their background. National Analysts also replaced the 
training program it initially used with a mentor system, in which newly 
hired coders were trained by having them code alongside a more experi- 
enced coder until they could perform satisfactorily alone. This varia- 
bility in both training and supervising large numbers of coders may 
have decreased the consistency with which individual questionnaires 
were coded. 


In July 1990, during our review, HNIS released the individual intake data 
for the NFCS basic sample in advance to several federal agencies so that 
they could analyze the information for policy decisions. However, in the 
preliminary documentation HNIS gave at least one federal agency, there 
was no reference to the survey’s low response rate other than that it 
was lower than expected. In August 1990, HNIS also announced plans to 
publicly release the individual intake data for the NFcs in September 
1990." 


In September 1990, concerned that HNIS was planning to release NFCS 
data without the appropriate disclosure of the low response rates, we 
wrote the Assistant Secretary for Food and Consumer Services that the 
1987-88 NFCS data might be substantially biased if the nonrespondents 
differed from the respondents. We urged that, with all future releases of 
the data for the basic and low-income samples, HNIS include information 
on the response rates and a discussion of the data’s potential bias. We 
also suggested how HNIS might investigate the potential bias, since 
National Analysts had lost the follow-up surveys of nonrespondents. 


In response, the Assistant Secretary stated that HNIS would 
assemble an independent expert panel to look at issues pertinent to bias, 
limitations in the survey’s use, and possible remedial actions; 


!1HNIS releases the data through the National Technical Information Service. 
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compare demographic distributions in the NFCS with those in census data 
using statistical significance tests; 

compare data from the NFCS with those from a comparable national 
health and nutrition survey conducted in the same period to determine 
whether they had similar results; 

compare response patterns of respondents contacted on the first 
attempt with those reached only after several attempts; and 

analyze and publish the results of the follow-up surveys of 
nonrespondents to characterize those who refused to participate. '” 


In November 1990, HNIS contracted with the Life Sciences Research 
Office of the Federation of American Societies for Experimental Biology 
(LSRO) to assess the impact of a low response rate on the dietary (indi- 
vidual intake) data from the 1987-88 NFCs’ basic sample and to make 
recommendations regarding any possible restriction on the data’s uses.'® 
In December 1990, we briefed the HNIS Associate Administrator on our 
preliminary findings of potential problems concerning the integrity of 
the NFCS data. LSRO convened an expert panel in February 1991 to con- 
duct its assessment. HNIS conveyed our findings to the panel for its con- 
sideration. LSRO and the panel examined the survey’s design and the 
execution of the NFCS; reviewed HNIS’ analyses, such as the comparative 
studies mentioned by the Assistant Secretary; identified additional anal- 
yses necessary to further evaluate the potential for bias due to the low 
response rate; and delivered a draft report in April 1991 on findings and 
critical issues regarding the potential bias." 


In its draft report, LsRo and the panel confirmed our findings that the 
survey was poorly executed, that the representativeness of the data was 
seriously limited, and that these problems needed to be fully disclosed to 
those who planned to use the survey. For example, the draft report 
agreed with our finding that the training and the monitoring of inter- 
viewers were insufficient. It also stated that the lack of data on 


!2National Analysts officials stated in writing to us that they had lost all records on the follow-up 
surveys of nonrespondents and would not deliver these data to HNIS. However, at that time HNIS 
officials said that National Analysts had told HNIS it was attempting to locate the records. Subse- 
quently, it was evident to HNIS that National Analysts was not likely to deliver these data. 


13 At the time, HNIS did not have LSRO and the panel examine bias in the low-income sample because 
National Analysts had not yet completed those parts of the survey. HNIS informed us that it is com- 


mitted to doing so, but would have to decide how to assess the integrity of the NFCS data for the low- 
income sample. 


14Impact of Nonresponse on Dietary Data From the 1987-88 Nationwide Food Consumption Survey, 
prepared for HNIS by LSRO Apr. 1991). 
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nonrespondents severely limited any attempt to compare the character- 
istics of respondents with those of nonrespondents. Without information 
on nonrespondents’ food consumption, the report concluded, there was 
no way to know whether weighting schemes or any other statistical 
adjustments could account for any differences between respondents and 
nonrespondents. 


The panel also examined HNIS’ test of the data’s representativeness 
caompaing 13 demographic characteristics with Census Bureau data. 
The panel found that the NFcS underrepresented ‘“‘nontraditional fami- 
lies” (for example, the survey included a smaller proportion of men and 
women from households headed by working females than from house- 
holds with non-working females). The panel also expressed concern 
about potential bias in HNIS’ reweighting scheme because of its extremely 
large range and unusual distribution of weights. Since nontraditional 
families were small in number in the sample, they were heavily 
weighted. However, the panel wrote, if they were not representative of 
the population’s nontraditional families, severe bias could result. 


The expert panel also agreed with our finding that the data might not be 
representative because of possible bias due to differences between 
respondents and nonrespondents (nonresponse bias). Its draft report 
stated, ‘It is certainly questionable whether or not the data provide 
unbiased estimates of the nation’s dietary intake.” As a result, the panel 
did ‘“‘not recommend use of the data from the 1987-88 NFcs”’ unless users 
employed the greatest caution. It questioned whether the data could be 
used for several kinds of estimates or analyses (such as estimates of 
specific foods or food groups, estimates of food intakes by small popula- 
tion subgroups, or trend analyses). 


In its draft report, the expert panel concluded, as we did, that HNIS must 
disclose the survey’s limitations in all releases of the data (data tapes) 
and in all publications reporting the data. The panel specifically urged 
that ‘‘a strongly worded cautionary statement concerning the potential 
for nonresponse bias” be used. At the time of our review, HNIS informed 
us that it planned to publish the data’s limitations, as described by the 
expert panel, in one public report. HNIS has prepared a disclosure state- 
ment for the data tapes, but plans were not yet completed for disclosing 
the data’s limitations in any technical reports.'® 


15 As of May 1991, HNIS had planned to publish one report on the household food consumption com- 
ponent of the 1987-88 NFCS and one report on the individual intake component. 
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The low response rate for the 1987-88 NFCS, the poor execution of its 
design, and the lax controls over the collection and processing of the 
data all raise serious doubts whether the survey meets its objectives—to 
evaluate the nutritional content of households and individual diets and 
detect shifts in food use since the previous survey was conducted. 


Nonetheless, the 1987-88 NFCs is the only survey available on household 
food consumption and individual diets over the last decade. Current 
data are critical for many users, including federal agencies that provide 
food assistance and regulate food safety. In our view, establishing confi- 
dence in the data will be difficult. HNIS does not have the key informa- 
tion needed to know whether, because of the low response rate and 
potential differences between nonrespondents and respondents in food 
consumption patterns, the survey’s results are representative of the U.S. 
population. 


Nor does HNIS know the extent of problems with the data. The expert 
panel convened by HNIS investigated only the basic sample. As yet, no 
one has scrutinized any of the data for the low-income sample. All data 
tapes and all publications reporting the data need to disclose the 
survey’s limitations so that federal officials and others can regard the 
survey critically before relying on it to make policy and program 
decisions. 


Meanwhile, similar problems in future surveys need to be averted. At 
least since the 1977-78 NFCS was conducted, HNIS has been aware that the 
lengthy questionnaire used to collect the household food consumption 
information places a burden on respondents and may discourage them 
from participating. Easing the burden on respondents and/or providing 
monetary or other incentives could achieve higher response rates and 
give those who use the survey greater confidence that the data are rep- 
resentative. Before the next survey in 1997-98, HNIS needs to correct 
flaws in the survey’s design and demonstrate that it can collect data 
that are timely and high in quality. Otherwise, HNIS will invite a repeat 
of problems experienced in the 1987-88 survey. 


To address problems with the quality of the 1987-88 NFcs data, we rec- 
ommend that the Secretary of Agriculture direct HNIS to 


disclose in all NFCS technical reports limitations that the expert panel 
finds in the NFCS’ basic sample data and 
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assess whether the data for the low-income sample were biased or other- 
wise poor in quality and, if so, disclose these limitations in all relevant 
technical reports. 


To minimize problems in future surveys, we recommend that the Secre- 
tary of Agriculture, before requesting funds for another NFCS, submit to 
the Congress a report 


demonstrating that HNIS has developed efficient survey instruments and 
procedures that reduce the burden on respondents, increase households’ 
motivation to participate in the NFCS, and meet essential data needs; 
describing a plan to ensure that the survey’s data will be representative 
of the U.S. population; and 

stating the steps to be taken and quality controls to be followed so that 
future surveys do not repeat the mistakes of the past. 


USDA recognizes the reality of many of the problems we identified and is 
taking action to correct them. Moreover, USDA agrees with GAO’s recom- 
mendations. (See app. I for details.) USDA expects to publish soon a 
report on the 1987-88 NFcs’ data problems, including the expert panel’s 
conclusions. USDA will also examine, though it has not yet determined 
how, whether the low response rate biased the low-income data, and 
will communicate its findings. 


USDA has initiated other actions to correct problems in the NFCs’ design 
and methodology. For example, HNIS has signed two interagency agree- 
ments under which the U.S. Bureau of the Census will help improve pro- 
cedures in HNIS’ food consumption surveys and assist in a variety of 
operations for future surveys. USDA agrees that a major reason for the 
NFCS’ poor response rate was the high level of burden on respondents. 
Officials stated that they are working to reduce this burden in all 
surveys. USDA canceled the planned 1992 Continuing Survey so that it 
could focus on incorporating the lessons learned from the 1987-88 NFCS 
into the planning, design, and execution of other surveys. 


usDA’s efforts appear to have potential for minimizing problems in 
future surveys, though we cannot yet evaluate whether they will be 
effective. Nonetheless, these are matters that USDA needs to fully 
describe in the report we recommend that it submit to the Congress. 
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USDA also commented on other more technical aspects of this report. We 
revised the report where appropriate and have included detailed 
responses to those comments in appendix I. 


Commenting on an excerpt from this report containing factual informa- 
tion, National Analysts wrote that the NFcs data were under review and 
meaningful information would be made available to government agen- 
cies and other users of these data. National Analysts also wrote, 
“Clearly, there were problems using a new data collection method in as 
demanding and complex a survey research effort as the Nationwide 
Food Consumption Survey. However, National Analysts attempted to 
address problems as they arose....”” Contrary to National Analysts’ 
views, the expert panel convened by HNIS confirmed our finding that the 
survey has serious limitations largely due to the low response rates. 
National Analysts’ poor execution of the survey probably further under- 
mined the data’s quality. For example, the training and the monitoring 
of interviewers were insufficient, and the lack of data on 
nonrespondents severely limited any attempt to compare the character- 
istics of respondents with those of nonrespondents. Without these data, 
there is now no way to determine whether the data are biased. 


National Analysts also commented on other more technical aspects of 


this report. We revised the report where appropriate and have included 
detailed responses to those comments in appendix II. 
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Contracting Practices 
Were Poor 


Planning Was Inadequate 


FNS and HNIS poorly managed the 1987-88 NFCS contract, at times circum- 
venting Key internal controls required to protect the federal government 
against fraud, waste, and abuse. Contracting and program office offi- 
cials poorly planned the acquisition and left out specifics in the contract 
defining the work to be performed and the deliverables due. The con- 
tract’s administration was so lax that the work National Analysts actu- 
ally undertook deviated substantially from the original contract’s 
requirements. The HNIS official who was designated the contracting 
officer’s representative (COR) frequently exceeded his authority by 
directing National Analysts to both forego certain contract requirements 
and undertake work not specified. The contracting officer remained 
unaware of the problems for much of the time. When he finally became 
aware of them, it was too late to reverse the damage done. The original 
contract’s budget—$6.2 million—was to cover costs between September 
1986 and March 1989, when the contract was to be completed. However, 
between August 1988 and June 1989, the contract’s costs increased to 
$7.6 million. Completion of the contract was delayed until April 1991. 


The planning and writing of a contract is important to ensure that the 
contractor is aware of what is needed and delivers the items on time at a 
reasonable cost. As the writer and provider of contractual documents, 
the government is generally held liable for any increased costs or delays 
the contractor incurs due to defects in a contract. Inadequate planning 
for the NFCS contract, as well as deficiencies in the contract’s specifica- 
tions, increased time and costs, delayed the delivery of required items, 
and prevented the performance of contract tasks and the recovery of all 
the government-owned lap-top computers National Analysts used to col- 
lect data. 


In several ways, HNIS did not adequately plan the NFCS contract; as a 
result, various problems occurred early on. For example, the 1987-88 
NFCS contract was awarded almost 9 weeks later than HNIS had planned. 
In its contract solicitation, HNIS planned to allow the contractor a 35- 
week period following the contract’s award to complete various tasks 
before it started interviewing households in April 1987. However, 
because of the delay in the contract’s award, HNIS reduced the time the 


‘As of May 1991, the NFCS contract was inactive but had not yet been closed out by the government 
and National Analysts. HNIS was reviewing final deliverables from the contractor to determine 
whether they were acceptable. 
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contractor was allotted for these early deliverables from 35 to 26 weeks, 
rather than extending the date planned for the start of interviewing. 


This new time pressure made the contractor’s work difficult, as did 
other delays. For example, although it was not the government’s fault, 
the competing company unsuccessfully protested the contract’s award 
to National Analysts. The protest slowed HNIs’ and National Analysts’ 
abilities to proceed for several weeks. Also, HNIS was not ready to work 
with National Analysts when the contract was awarded and the com- 
pany was authorized to start work. For example, HNIS needed to give 
National Analysts codes for foods so that the company could program 
the household questionnaire and the individual intake food lists by 
December 1986. HNIS did not give all of the necessary information to 
National Analysts until after March 1987. HNIS also required National 
Analysts to prepare printed versions of draft NFCS questionnaires by 
November 1986. HNIS was to use those documents in applying the same 
month for OMB’s approval of the survey.? Although National Analysts 
delivered the documents on time, HNIS then made revisions that National 
Analysts had to incorporate. This delayed HNIs’ application to OMB. As a 
consequence, OMB did not receive the application until January 1987. 
OMB approved the survey in March 1987, 2 months later than scheduled 
under the contract. 


This delay, in combination with the reduced lead-time before data collec- 
tion was to begin, led to the omission of a major contract task—the 
dress rehearsal. National Analysts was to conduct the dress rehearsal in 
January 1987, report the results to HNIS, and incorporate necessary 
changes in the survey’s operations by the time National Analysts was 
scheduled to start collecting data in April 1987. National Analysts could 
not conduct the dress rehearsal until its computer programs were com- 
plete and OMB approved the survey; consequently, National Analysts and 
the cor dropped the requirement. 


The NFCS contract did not adequately identify requirements. For 
example, the original contract did not name all the deliverables HNIS 
needed, such as data files on the surveys of nonrespondents, data files 
containing the screening forms, the handbook for post-field procedures 
used to process the data, and documents on statistical procedures. 


2Under the Paperwork Reduction Act of 1980 (P.L. 96-511), federal agencies must obtain OMB’s 
approval for all surveys involving 10 or more respondents. OMB is allowed up to 90 days to act on 
each application. 
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Because of the original contract’s vagueness or omissions, HNIS and FNS 
were confused about what deliverables were due, even though subse- 
quent modifications attempted to specify them. The contract also had to 
be modified because HNIS more specifically defined the deliverables to 
“clarify performance issues,” according to FNS’ contract management 
branch chief. According to the HNIS official who took over as COR in 
October 1990, the original contract did not contain all the details 
because HNIS had assumed that National Analysts would know what was 
required. She also stated that contracts lacking similar details had been 
used successfully with National Analysts on earlier food consumption 
surveys. 


The contract also did not contain adequate instructions to control 
National Analysts’ return of the lap-top computers used to administer 
the questionnaire on household food consumption. As a result of this 
and the contractor’s poor inventory control, government property was 
lost, damaged, or stolen, or returned long after National Analysts had 
finished using it. With funds from the contract, National Analysts had 
purchased 210 lap-top computers for $338,000. Although the contract 
stated that these computers would remain HNIS’ property, it did not 
specify how National Analysts would dispose of the property once it 
was no longer being used or when it was to be returned to HNIS. The 
contracting office did not review or approve National Analysts’ prop- 
erty control procedures, nor did National Analysts adhere to them. 
Because of other concerns, the contracting office did not ask National 
Analysts about the computers until March 1989, 12 months after the 
company was supposed to have finished using the equipment. Seven 
months later, National Analysts disclosed for the first time that 34 of 
the 210 lap-tops had been lost, damaged, or stolen. For almost 1 year— 
between October 1989 and September 1990—the contracting office tried 
to recover the lap-tops.? The contract management branch chief stated 
that the normal procedure of any government agency is not to dispose of 
the property under a contract until the contract is ended or until other- 
wise instructed. 


3National Analysts paid FNS $10,200 for the missing units, based upon FNS’ market survey of cur- 
rent value of the units in 1990. The original cost was almost $55,000. 
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The administration phase of the procurement process begins with a con- 
tract’s award, continues through the contract’s implementation and com- 
pletion, and ends with final settlement and payment to the contractor. 
During this phase, the government ensures that the work is done 
according to the contract’s requirements and that the contractor is paid. 
Although other factors contributed to the survey’s delays and cost over- 
runs, deficiencies in the administration of the NFCS contract exacerbated 
these problems and prevented HNIS from receiving completed contract 
items. These deficiencies violated internal controls for contracting, 
which are designed to protect the public’s interest. 


COR Exceeded His 
Authority 


The cor was responsible for monitoring the performance of National 
Analysts and providing technical assistance. He was not authorized to 
change the contract’s work, costs, or completion dates or to enforce the 
contract’s provisions. This authority was vested only in the contracting 
officer—as an integral part of internal controls for contracting. How- 
ever, the cor repeatedly bypassed the contracting officer, interfering 
with the contracting officer’s responsibility for negotiating a fair cost 
for the work before it was started.* 


The cor exceeded his authority in several ways. He waived the require- 
ment for National Analysts to deliver computer programs for HNIS’ 
approval by December 1986; HNIS statistical staff were requesting the 
programs as late as April 1988. He and National Analysts negotiated to 
drop the requirement for the dress rehearsal. In July 1987, they negoti- 
ated the elimination of the seasonal sampling for the household inter- 
views, and with that, dropped requirements for operations reports and 
data tapes on quarterly progress. In January and February 1988, the 
cor and National Analysts negotiated a 5-month extension in data collec- 
tion, beyond the contract’s 12-month requirement. 


COR’s Monitoring and 
Communications Were 
Poor 


The cor, as the contracting officer’s eyes and ears, is required to monitor 
the technical progress of the contractor and, when a problem arises, 
notify the contracting officer, who can enforce the contract. However, 
the cor for the NFCS contract poorly monitored National Analysts’ per- 
formance. For example, until we informed him in August 1990, he said 
he was unaware that National Analysts had not processed the results of 


4 As requested by Representative George E. Brown, Jr., our Office of Special Investigations is investi- 
gating the 1987-88 NFCS contract to determine whether any conflicts of interest existed between 
USDA personnel and the contractor and whether charges submitted by the contractor were proper. 
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the follow-up surveys of nonrespondents before losing the records in a 
January 1990 office move. He said he did not monitor National Ana- 
lysts’ performance of these surveys in late 1988 and did not know 
whether the company actually performed the work. Although he said he 
frequently communicated with National Analysts about the contract, 
the cor kept few written records and did not document most of the tech- 
nical guidance and directions that he gave National Analysts. 


The cor did not communicate with the contracting officer. The cor did 
not notify the contracting officer about problems with low response 
rates or his directions to National Analysts to deviate from the con- 
tract’s requirements for collecting data. The CoR assumed that the con- 
tracting officer and his staff read National Analysts’ monthly progress 
reports for details about technical problems. 


For a number of years, there were several different HNIS administrators. 
During most of that time, according to one former administrator, the COR 
was responsible for the agency’s budget and contracts. Two former HNIS 
administrators said that they were unaware of the cor’s role in misman- 
aging the NFCS contract. These previous administrators told us that they 
had little or no knowledge of the NFcs’ technical requirements and 
assumed that the COR was responsibly managing the contractor because 
of his many years of experience. We also discussed the contract’s 
problems with HNIS’ associate administrator (who arrived in May 1988 
and was acting administrator from November 1988 through April 1990) 
and informed him that the cor had improperly directed National Ana- 
lysts to deviate from the contract’s technical requirements over an 
extended period of time and was not adequately monitoring the con- 
tract. According to the current administrator (who arrived in May 1990) 
and associate administrator, the Cor’s managerial style and lack of 
recordkeeping had made it difficult for them to learn what was going on 
with the NFCS contract. However, the associate administrator said that 
he actively participated in efforts to resolve the contract’s problems, 
including converting the contract to a fixed-price basis in June 1989 to 
avoid further cost overruns. After we disclosed the management 
problems described above, the HNIS administrator relieved the cor of his 
responsibilities in October 1990 and appointed another HNIS official as 
COR for the NFCS contract. 
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Contracting Officer Did 
Not Monitor the NFCS 
Contract 


The contracting office is responsible for ensuring that all necessary 
actions are performed so that a contract is effective, for ensuring com- 
pliance with the contract’s terms, and for safeguarding the interests of 
the United States in its contractual relationships. During a discussion 
with FNs officials in August 1990, FNS’ contract management branch 
chief said that when he arrived in early 1988, he recognized that at FNS 
there was basically no contract administration going on with the 1987- 
88 NFCS. He also agreed that the cor and the contracting office did not 
communicate during the contract’s critical first 17-month period, though 
they should have. Lower-level contract specialists assigned to the NFCS 
contract during this time stated that they did not discuss the contract’s 
problems with the cor. According to the branch chief, these practices 
are not consistent with the branch’s current practices. 


The contracting office also did not detect or act on early signs of per- 
formance problems. Although the contracting office received National 
Analysts’ monthly progress reports, officials depended on the Cor to 
alert them to any problems. The contracting office also did not investi- 
gate why National Analysts had spent 75 percent of the total budget by 
April 1988, almost 1 year before the contract was to end. Under the limi- 
tation-of-cost clause, a contractor is required to notify the contracting 
officer when the costs the contractor expects to incur in the next 60 
days, when added to costs previously incurred, will exceed 75 percent of 
the contract’s funds. Neither the government nor the contractor is obli- 
gated to reimburse or incur costs in excess of the estimated costs until 
the contracting officer acts to increase the contract’s budget. National 
Analysts did not notify the contracting officer until July 1988. The 
branch chief stated in March 1991 that although the contracting office 
should have recognized the spending level without the contractor’s noti- 
fication, upon notification, the office took quick action to resolve the 
issues. 


FNS officials currently involved with the NFCS contract agreed that the 
contracting office’s monitoring was poor in the past. The contract man- 
agement branch chief cited several factors, including a heavy work load 
of contracts, different management style and operating practices by his 
predecessor, staff turnover, and the assignment of inexperienced con- 
tract specialists. Since his appointment in 1988, he has attempted to 
resolve problems with the NFCS contract, including changing it to a fixed- 
price contract to avoid further cost overruns. However, his actions were 
too late to correct the improper changes in the contract that the Cor had 
allowed. Contracting officials noted that they had no financial leverage 
to improve performance now because National Analysts had received 
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most of the payment owed under the contract. Nor was it practical, 
according to contracting officials, to terminate the contract for default, 
since that would delay a congressionally mandated study. They also 
stated that USDA could not recover any of the money paid to National 
Analysts because of its poor performance. This was partly because the 
contract was a cost reimbursement type contract, but also because USDA 
had technically mismanaged the contractor’s work. Thus, USDA would 
probably lose any legal dispute with National Analysts to recover the 
money. 


In addition to negotiating improper changes in the contract work with 
the cor, National Analysts contributed in other ways to deficiencies in 
the NFCS contract. For example, since June 1988, National Analysts has 
been unable to deliver the data tapes and other deliverables on time, 
despite the fact that five contract modifications incorporated National 
Analysts’ own schedules for delivery. National Analysts also assigned 
interviewing and data processing staff who were inexperienced, poorly 
trained, and unable to do the work. It used workers from temporary 
employment agencies for the data processing, which was not provided 
for under the contract. By departing from planned personnel assign- 
ments, National Analysts lost control of the contract’s management 
early in the project. The project director and assistant project director, 
who were Key personnel responsible for day-to-day management of all 
operations, cut their hours on the project starting in August 1987. The 
assistant project director stopped charging hours on the project in 
November 1987, while the project director left the project in September 
1988. 


The mismanagement of the 1987-88 NFcs contract at nearly every 
stage—from planning to administration—and by all parties con- 
cerned—HNIS’ COR, FNS’ contracting officer, and National Analysts—had 
deleterious consequences. The contract, scheduled for completion in 
March 1989 at a cost of $6.2 million, has had cost overruns amounting to 
$1.4 million, and only a portion of the survey’s data has been released to 
date. It is difficult to predict when the complete survey will be available 
to the public, and as the previous chapter showed, even then its data 
may be questionable. The delay and poor quality are yet further conse- 
quences of the improper changes in the contract that the Cor and 
National Analysts agreed to. Furthermore, in violating internal controls 
that were in place to safeguard the government’s best interests, officials, 


Page 35 GAO/RCED-91-117 USDA’s Nationwide Food Consumption Survey 


oe SS SS SS eee eee ee 


Chapter 3 
Management of NFCS Contract Was Deficient 


Recommendations 


Comments From the 
Agency and the 
Contractor and GAO’s 
Response 


at a minimum, wasted federal funds. More attention is needed in plan- 
ning and writing contracts so that all requirements are specified and 
adequate internal controls are identified, for example, in the handling of 
government property. 


We recommend that the Secretary of Agriculture direct FNS and HNIS, in 
contracts for future surveys, to ensure that existing contracting proce- 
dures are followed and people are held accountable for carrying out 
their responsibilities. Particular attention should be given to 


contract planning, 

contract writing, 

compliance with the contracts’ terms, and 

contract administration and monitoring by Cors and contracting officers. 


USDA recognizes the reality of many of the problems we identified and is 
taking action to correct them. Moreover, USDA agrees with our recom- 
mendations. (See app. I for details.) For example, USDA plans to improve 
its monitoring of contracts by visiting the contractor’s central office 
weekly, having a formal survey operations team assist the Cor, and com- 
mitting additional staff with in-depth contracting experience. These 
actions appear to have potential for implementing our recommendation 
that existing contracting procedures be followed, though we cannot yet 
evaluate whether they will be effective. 


USDA also commented on other more technical aspects of this report. We 
revised the report where appropriate and have included detailed 
responses to those comments in appendix I. 


National Analysts commented on an excerpt from this report containing 
factual information. It wrote, “Clearly there were problems using a new 
data collection method in as demanding and complex a survey research 
effort as the Nationwide Food Consumption Survey. However, National 
Analysts attempted to address problems as they arose and performed 
the requirements of its contract. Contrary to National Analysts’ views, 
this report documents that the contractor did not perform all the con- 
tract’s requirements. For example, the requirement that National Ana- 
lysts conduct a dress rehearsal was waived by National Analysts and 
the cor, not by the contracting officer, as the contract required. National 
Analysts also did not deliver any data from the required follow-up 
surveys of nonrespondents. These omissions not only violated contract 
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requirements but also had serious negative effects on the quality of the 
survey. 


National Analysts commented on other more technical aspects of this 


report. We revised the report where appropriate and have included 
detailed responses to those comments in appendix II. 
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Note: GAO comments 


supplementing those in the [ae ae ae 
report text appear at the 


end of this appendix. United States Human Nutrition Belcrest Road 
Department of Information Hyattsville, Maryland 
Agriculture Service 20782 


May 9227, 199) 


SUBJECT: Comments on General Accounting Office Draft Report 
RCED-91-117, "FOOD ASSISTANCE: Mismanagement of 
Nutrition Survey Has Resulted in Questionable Data" 


TO: John W. Harman, Director 
Food and Agriculture Issues, Resources, 
Community, and Economic Development Division 
General Accounting Office 


Attached are the Department of Agriculture’s comments on the 
subject report. This attachment reflects the views of all 
agencies within the Department of Agriculture having 
responsibilities related to the subject matter of the draft 


report. 
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The Department recognizes the reality of many of the problems 
identified during the GAO review of the 1987-88 NFCS, and are 
taking action to correct them. We agree with the recommendations 
offered by GAO. Section I below highlights these actions. 


There are, however, some statements in the draft report that we 
believe need correction or clarification. Section II below 
discusses these areas of disagreement. 


Section I: Actions Taken by HNIS and FNS to Correct Problems 


The GAO review focused on two areas: survey design, methodology, 
and execution; and contract management. We have taken steps to 
address weaknesses in both areas. 


A. Survey Design, Methodology, and Execution 


1. HNIS has signed two interagency agreements with the U.S. 
Bureau of the Census. One agreement is with the Center for 
Survey Methods Research. HNIS will be working closely with the 
Center in assessing and improving procedures in our ongoing 
Continuing Survey of Food Intakes by Individuals (CSFII). In 
particular, the Center will be evaluating the survey instruments 
-- screener questionnaire, nonresponse questionnaire, household 
and dietary intake questionnaires, instruction booklet, and 
interviewer procedures -- and making suggestions for improvement. 
The Center will also be conducting or advising on cognitive and 
experimental research on technical issues such as portion-size 
estimation, use of surrogate respondents, and number and spacing 
of interviews to measure dietary intake. 


The second agreement is with the Bureau’s Demographic Surveys 
Division to enlist their help in developing Requests for 
Contracts regarding sample design and survey management reports; 
to provide assistance relating to data collection, data 
processing, and quality control; to advise on evaluations of 
nonresponse, adjustment for nonresponse, and sample 
poststratification; and to provide continuing support for HNIS 
survey operations. 


2. In 1988, HNIS signed a contract with the University of Texas 
to work with us on developing a survey data management system 
which would a) provide more flexibility in coding food recipes 
than had been previously available, b) enhance documentation and 
quality control over existing systems, and c) be available to all 
potential survey contractors, thus allowing greater competition. 


3. HNIS has developed and is putting into place a contract to 
provide for independent review and evaluation of issues related 
to USDA's nutrition monitoring activities. This contract will 
allow us to obtain state of the art, authoritative, independent 
reviews and evaluations of selected current and emerging issues 
related to our survey design, execution, analysis, and reporting. 
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4. HNIS has detailed three professional staff members into the 
Food Consumption Research Branch of HNIS and has specified 
monitoring our survey contracts and providing quality control for 
the data as the top priority in the Agency. 


5. HNIS agrees with GAO that a major reason for the poor 
response rate in the NFCS was the high level of respondent 
burden, due in part to conducting the household and individual 
surveys together. HNIS is working to reduce the respondent 
burden in all of our surveys. For example, the USDA Food Plans 
were originally developed using both individual and household 
components from the same survey. However, as part of a larger 
program to re-evaluate our data needs, HNIS is currently 
exploring alternatives for the development of the Food Plans. 
This is only one example of our efforts to reduce respondent 
burden. 


6. In February of 1999, HNIS tentatively decided to cancel the 
1992 CSFII in order to have the opportunity to incorporate the 
lessons learned from the NFCS experience and the above research 
activities into our survey planning, design, and execution. We 
discussed these plans with our major Federal users, the Food and 
Nutrition Service, the Food and Drug Administration, and the 
Environmental Protection Agency. After receiving their 
concurrence, we further discussed our plans with personnel of 
appropriate Congressional program committees, with our 
Appropriations Committees, with OMB, and with non-Federal data 
users and interested organizations. We believe we have consensus 
that this plan is acceptable and appropriate and it has now been 
implemented. 


B. Contract Management 


1. The GAO review identified weaknesses in the performance of 

See chapter 3. the Contracting Officer’s Representative (COR) for the NFCS. We 
believe this may have resulted in part from lack of recent 
refresher training in the duties and responsibilities of the COR. 
HNIS has signed a contract to bring such refresher training to 
all professional HNIS personnel and to provide introductory 
training to new staff. 


2. HNIS has changed its organization to provide that the COR of 
major survey contracts will not be burdened by other 
responsibilities, including supervisory responsibilities. The 
COR is to focus full-time upon monitoring the contract, report 
directly to the Director of the Nutrition Monitoring Division and 
have immediate access to the Director. 


3. FNS has loaned HNIS a specialist to review the existing 
contract for the 1989-91 CSFII and to assist in developing the 
Request for Contract (RFC) for the 1993-96 data collection. This 
effort is aimed at introducing more and better management 
controls into HNIS’s contract monitoring. 


Page 40 GAO/RCED-91-117 USDA’s Nationwide Food Consumption Survey 


Now on pp. 12 and 26. 
See comment 1. 


Appendix I 
Comments From the Department 
of Agriculture 


4. For the survey currently in the field, the 1991 Continuing 
Survey of Food Intakes by Individuals, the COR and other senior 
HNIS staff are making weekly trips to the contractor’s central 
office to monitor progress on the survey and identify problems at 
an early stage. A Survey Operations Team has been established 
within HNIS, which also includes representatives from the Census 
Bureau and the FNS contracts officer, to provide assistance to 
the COR in monitoring the contract. Team meetings are held on a 
weekly basis. 


5. Additional resources have been committed by FNS to the 
improvement of the capabilities of the Contracts Management 
Branch. New staff hired over the past three years have had more 
indepth contracting experience in all types of contracts. The 
level of training has been increased and a majority of the staff 
meet the level III and IV requirements to be warranted under the 
USDA guidelines requiring at least 400 hours of training. The 
office has been automated and programs developed to track each 
contract from the planning stage through closeout. Before the 
beginning of each fiscal year a contracting plan is developed for 
the entire year. Each requirement is assigned to a contract 
specialist who begins working with the COR even before the 
development of a Request for Contract. A standard procedure has 
been developed to assist the CORs in preparing their Requests for 
Contracts. The team concept in contracting is followed 
throughout the entire life of the contract. 


6. Electronic communication between FNS and HNIS has been 
designed to improve communications on contract matters. Reports 
are submitted to management on a weekly basis from the automated 
system along with a summary of concerns which might require their 
involvement. This has been done with a reduction in the 
authorized staffing level. Every effort is being made to provide 
more and better contracting support for managing the agencies’ 
programs. 


Section II. Areas of Disagreement Between GAO and HNIS/FNS 


1. In the introduction to Chapter 2 and again in the 
conclusions of that chapter (pp. 13 and 32 of the draft report), 
GAO stresses the need for HNIS to disclose the limitations of the 
1987-88 NFCS in technical reports. HNIS agrees with this 
recommendation. In the documentation accompanying the survey 
data tape first released to the public on October 1, 1990, we 
drew attention to the low response rate and the potential for 
nonresponse bias and stated that further information would be 
shared as it became available. We have now completed our in- 
house studies of the impact of nonresponse and provided the 
results to the Expert Panel convened by the Life Sciences 
Research Office under contract with HNIS to examine the integrity 
and validity of NFCS data. 
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We anticipate publication shortly of our report on nonresponse in 
the 1987-88 NFCS, which will include summaries of our studies on 
nonresponse; a review and recommendations by Dr. Wayne Fuller, 
our statistical consultant; the report of the Expert Panel; and 
conclusions and recommendations by HNIS. 


The attached two-page statement, "1987-88 Nationwide Food 
See comment 2. Consumption Survey (Individual Data): Nonresponse Issues," will 
accompany all releases of data from the NFCS. It has been 
provided to Federal agencies and researchers who have purchased 
or been sent the data tape. We plan to respond to requests for 
data from the NFCS only in written form (mail or fax) and to send 
the statement along with the requested data. 


2. With regard to GAO’s discussion of the design of the NFCS on 
p. 16 cf the draft report, a clarification of the reasons for 
See comment 3. collecting multiple days of intake information from individuals 
in HNIS surveys is needed. The scientific community has 
repeatedly recognized the need for obtaining multiple days of 
intake for certain purposes. This was reaffirmed in a 1984 
report from the National Academy of Sciences that recommended 
that HNIS continue to collect at least three days of dietary 
intake data (National Survey Data on Food Consumption: Use and 
Recommendations," National Academy Press, 1984). Methodological 
research has shown that data from one day support the reporting 
only of mean intakes; multiple-day data are necessary to derive 
estimates of the distribution of intakes among individuals within 
groups and to capture day-to-day variances in food consumption. 
Therefore, both the NFCS and CSFII are designed to capture three 
days of dietary intake data. 


Now on p. 16. 3. The description on p. 18 of the draft report of the list- 
S mA recall procedure in the household food use component of the NFCS 
ME STINDS HES is misleading. GAO implies that the method, used in the 1977-78 


and 1987-88 surveys, requires the respondent to listen to the 
reading of 350 main food groups and subgroups. While these food 
groups and subgroups appeared on the computer screen (in the 87- 
88 survey), the interviewer instructions directed them to read 
only the main food groups and if no foods were used in that group 
to skip to the next main group. This reduces the burden 
considerably. Nevertheless, HNIS will reexamine the household 
food use methodology to determine ways to reduce the length of 
the interview without sacrificing data quality. 


GAO's suggestion to reduce the food items to those with 
nutritional differences may not be appropriate. Many of the 
applications of the NFCS data are concerned with food use rather 
than with nutrient availability. Exploration of possible 
reductions in the number of food subgroups must consider the 
needs of the data users and not merely focus on nutritional 
differences. 
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Now on p. 16. 


See comment 5. 


Now on p. 15. 


See comment 6. 


Now on p. 17. 


See comment 7. 
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4. The draft report, in critiquing the NFCS methodology, twice 
states (pp. 4 and 19 of the draft report) that the individual 
response rate in the 1977-78 NFCS was 43 percent, based on a 
household response rate of 61 percent and a response rate of 70 
percent for individuals within households. This is in error: 
GAO computed the response rate on the assumption that all 44,169 
individuals in participating households should have been 
interviewed. However, the sampling design for the 1977-78 NFCS 
called for interviewing all individuals only in the first 
quarter; for the remaining three quarters only 50 percent of 
adults in multi-adult households were to be interviewed. The 
number of eligible individuals was thus 32,803 rather than 
44,169. Since 30,770 individuals provided usable data, the 
within-household individual completion rate was 94 percent and 
the net individual response rate was 57 percent. 


5. On p. 19, the draft report states that HNIS changed its 
approach to food consumption surveys following the 1977-78 NFCS. 
The methodology used for the NFCS 1987-88 was very similar to 
that used in 1977-78. However, in the decade between these 


surveys, HNIS introduced the Continuing Survey of Food Intakes by 


Individuals (CSFII) to respond to the need for yearly or 
continuous data on dietary status. In the 1985 and 1986 CSFII, 
following a recommendation from a National Academy of Sciences 
panel, HNIS surveyed individuals on six nonconsecutive days 
rather than 3 consecutive days as in NFCS. 


6. In critiquing HNIS’s introduction of computer-assisted 
personal interviewing (CAPI) in the 1987-88 NFCS, GAO implies 
(p. 20 of the draft report) that the sole rationale for this 
innovation was to shorten the time needed for data processing. 
This goal was explicitly stated in the Request for Proposals 
because a reduction in data-processing level of effort was 
expected to be reflected in the proposals submitted by offerors. 
However, there were a number of additional reasons why HNIS 
wished to use CAPI. By saving on writing time and paper 


shuffling, it was expected to reduce the length of the interview. 


It was expected to reduce the time between the interview and 


possible callbacks for clarification of responses, thus improving 


the willingness and ability of respondents to answer callback 
questions. Most important, it was expected to improve the 
quality of the data by reducing transcription errors and 
interview errors such as mis-sequencing of questions, improper 
application of skip patterns, or recording of erroneous codes. 
HNIS believes that, while there were indeed some problems with 
this first attempt at using CAPI, on the whole the use of CAPI 
has the potential for improving data quality. 


xia 
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Now on pp. 20-21. 
See comment 8. 


Now on pp. 26-27. 
See comment 9. 


Now on pp. 29-30. 
See comment 10. 
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7. GAO’s description (pp. 25 and 31 of the draft report) of the 
weighting applied to the individual intake data contains several 
errors. The discussion implies that weights were constructed for 
day of the week and for each season, and then additional weights 
were applied to adjust for demographic characteristics. In 
reality, a single set of weights was constructed based on least- 
squares theory, such that their use removes any bias that is 
linearly related to the control variables, which included 
variables for day of the week and month of the year, the four 
geographic regions, and for such demographic characteristics as 
household composition, household income, employment status of the 
heads of household, race, age, and sex. 


In addition, the weights are integers rather than ratios, and 
their range is 1 to 136. The mean of the weights is 23.5, which 
should be noted since it is the magnitude of the large weights 
relative to the average that affects the variance of the 
differential weight procedure relative to the constant weight 
procedure. While the use of these weights indeed results in 
higher overall sampling errors, weight ranges of this magnitude 
are common in national surveys. The ratio of the maximum to 
average weight in the 1987-88 NFCS is 5.8; for comparison, it is 
6.2 in the March 1987 Current Population Survey and 11.9 in the 
1987 National Health Interview Survey. 


8. On pp. 31-32 of the draft report, GAO recommends that the 
Expert Panel convened to examine the impact of nonresponse 
investigate both the basic and low-income surveys. The Expert 
Panel was asked to examine only the data for the basic sample. 
This decision was made in the interest of timeliness: HNIS 
believes that it is critical that the Expert Panel’s assessment 
be made and communicated quickly since many users have urgent 
needs for the data. Waiting for the low-income data would have 
delayed an assessment for at least a year. HNIS is committed to 
examining and communicating the impact of nonresponse on the low- 
income data, but has not yet determined what mechanism will be 
most appropriate to obtain this examination. 


9. The draft report states (pp. 34 and 35) that there was 
inadequate planning for the acquisition. The file contains 
evidence of extensive acquisition planning as required by the 
Federal Acquisition Regulation. The 1987-88 NFCS contract is 
identified on the Advanced Acquisition Plan for FY 1986. The 
file contains: a) an approved Acquisition Procurement Plan for 
this contract dated, March 3, 1986; b) necessary Contracting 
Officer Determination & Findings dated, April 14, 1986; c) a 
preproposal conference attended by 12 contractors held on May 16, 
1986; d) Office of the General Counsel review dated May 29, 1986, 
and e) Equal Opportunity Office review dated September 25, 1986. 
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Now on p. 29. 10. On p. 34, the draft report states that as of January 1991, 
contract costs had jumped to almost $8 million (actually $7.6 
million) implying frequent and recent cost increases. The last 
time additional funds were made available to cover contract costs 
was June 9, 1989 with modification 5. The procurement was 
originally solicited, negotiated, and awarded as a Cost Plus 
Fixed Fee contract. Modification 5 increased contract funding by 
$468,929, an amount previously recognized as necessary to 
complete the contract. This bilateral modification also 
converted the NFCS contract to Firm Fixed Price specifically to 
contain costs and avoid additional cost overruns which would 
accrue to the Government. At that time, all increased cost and 
funding risks shifted from the Government to National Analysts. 


See comment 11. 


The draft report highlights the fact that a cost overrun 
occurred. The report should not ignore the root causes: a) the 
government’s requirement to complete the survey and the need to 
incur all necessary costs to accomplish this objective; and b) 
the survey and analysis took more work than originally 

Now on p. 35. anticipated. On p. 42, the draft report suggests that 
termination for default was not a practical solution "according 
to contracting officials ... since that would delay a 
congressionally mandated study." In fact, the contracting office 
presented termination for default as a business option for 
dealing with a contractor who was repeatedly late in meeting 
scheduled deliverables. This business judgment was overridden by 
the desire of Congress to receive the information for which the 
contract was written. 


11. HNIS agrees with GAO that proper contract management 
procedures were not followed when the decision was made to drop 


Now on p. 32. the planned dress rehearsal (p. 36 of the draft report). 
S 12 However, we believe it is useful to clarify the reasons for 
AMUSE j dropping this requirement. Survey preparations were delayed for 


numerous reasons, some of which are attributable to HNIS and 
others which are not. The survey was scheduled to begin on April 
1, 1987. On that date, the dress rehearsal had not been 
conducted and there was concern that delaying the survey would 
have increased the cost substantially. The contractor would have 
had to continue to pay the interviewers and coders that had 
already been hired or lay them off and hope to rehire them at a 
later date. The COR believed that the dress rehearsal could be 
omitted for several reasons: The contractor had turned in a 
successful performance in the 1985 and 1986 CSFII (individual 
intake) and the 1984 Puerto Rican Nutrition Survey (household use 
of food). In addition, the contractor has over five decades of 
survey research experience including over 20 food consumption 
research studies. It has conducted research for numerous other 
Government agencies and has a proven track record. The survey 
team at National Analysts included the same senior individuals 
who had successfully directed previous food consumption surveys 
for HNIS. In retrospect, the decision to drop the dress 
rehearsal was the wrong decision. However, it was a decision 
that at the time was made for what was believed to be the best 
interests of the Government. 
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Now on p. 31. 


See comment 13. 


Now on p. 34. 


See comment 14. 


Now on p. 34. 


See comment 15. 
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12. On p. 37, the draft report questions administrative 
practices with respect to property management. Instructions 
regarding property, in the form of required Federal Acquisition 
Regulation (FAR) clauses, were incorporated into the contract. 
That National Analysts lost 34 laptop computers is not the result 
of poor instructions from the government, but rather inadequate 
handling of the property by the contractor. Additional 
instructions regarding disposal of property were not required. 
The normal practice is to recover property at the conclusion of 
the contract. Full accountability for all government property 
has been accomplished. 


13. GAO states that the contracting office is responsible for 
ensuring that all necessary actions are performed for effective 
contracting, etc. (p. 40 of the draft report). We agree with 
this and add the following comment. Although the Contracting 
Officer has overall responsibility for the contract, it is 
impossible for him/her to be knowledgeable in the many technical 
areas for which contracts are written. Survey contracts have 
many unique features. Methodologies, data, and design 
requirements are generally outside the specific knowledge of the 
Contracting Officer. The Contracting Officer must rely on the 
abilities and skills of the program staffs and more specifically 
on the Contracting Officer’s Representative. In the course of 
developing contracts, disagreements naturally arise between the 
Contracting Officer and the program staff over the scope of work, 
deliverables and other factors. Often the Contracting Officer 
will defer to the program staff since it possesses the requisite 
technical knowledge. 


14. On p. 40, the draft report states that the FNS’s contract 
Management branch chief "agreed there was no communication 
between the COR and the Contracting office during the contract’s 
critical first 17-month period." This is incorrect. The branch 
chief recognizes that the file does not contain any records of 
correspondence or conversations pertaining to general contract 
administration. However, during this time, invoices and other 
modifications were processed. Extensive contract oversight did 
take place during 1988 and beyond. The ultimate result of this 
contracting officer involvement was the recognition of the 
problems with the contract and the conversion to a Firm Fixed 
Price contract. This action occurred well before GAO conducted 
the audit of the National Food Consumption Survey contract. 
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United States Human Nutrition Belcrest Road 
\} Department of Information Hyattsville, Maryland 
47 ~~ Agriculture Service 20782 


1987-88 Nationwide Food Consumption Survey 
Individual Intake Data 
Nonresponse Issues 


A. Response Rates 


The response rates for the 1987-88 Nationwide Food Consumption 
Survey (NFCS) were very low, approximately 38 percent at the 
household level and 31 percent at the individual level. This 
means that about three of every eight occupied households 
selected for interviewing were contacted and participated. 
Within these households, 81 percent of the eligible individuals 
provided at least one day of intake data. 


Household and individual participation results were as follows: 


Housing units selected: IL Sh e/ Sie) 

Occupied housing units: 12,181 (89% of housing units selected) 

Contacted households: 9,935 (82% of occupied housing units) 

Screened households: 8,450 (69% of occupied housing units; 
85% of contacted households ) 

Participating households: 4,589 (38% of occupied housing units; 
54% of screened households ) 

Individuals in partici- 


pating households: 2 Fo 2 
Individuals completing 
Day 1 recall: 10,172 (81% of individuals in partici- 


pating households; estimated 31% 
of individuals in all occupied 
housing units) 
Individuals completing 
3 days recall/records: 8,468 (83% of individuals completing 
Day 1 recall) 


B. Weighting Factors 


If respondents and nonrespondents have systematically different 
behavior, then survey results may be biased. Comparison of 
individuals who participated in the 1987-88 NFCS with estimates 
from the Census Bureau’s March 1987 Current Popuiation Survey 
revealed slight underrepresentation of a number of demographic 
groups, including employed female heads of households and higher- 
income households. Further, unequal numbers of interviews were 
obtained in different calendar months and on different days of 
the week. For these reasons, the data were weighted to yield 
estimates that match the population distribution of 13 
demographic characteristics which are related to food intake 
behavior and to equalize interviews over months of the year and 
days of the week. While this weighting is expected to reduce the 
magnitude of nonresponse bias, it also increases the variance of 
the estimates. 
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Cc. Cautions Regarding Increased Variance of Estimates 


For any survey employing a complex sampling design it is 
recommended that measures of variance be estimated using 
specialized software (such as OSIRIS, PC/CARP, SUDAAN or 
SESUDAAN) which takes survey design and weighting into account. 
This is of particular importance with 1987-88 NFCS data because 
of the increased variance resulting from the weighting. It is 
essential that estimates of variance such as standard errors and 
coefficients of variation be evaluated for all variables of 
interest. 


HNIS policy is to identify estimates for which the coefficient of 
variation is between 25 and 50 percent and those with 
coefficients of variation greater than or equal to 50 percent. 
Estimates with coefficients of variation greater than or equal to 
50 percent should not be used or reported alone. We urge that 
these guidelines be followed by others reporting NFCS data. 


D. Cautions Regarding Nonresponse Bias 


An independent Expert Panel convened to assess the impact of 
nonresponse in the NFCS concluded, and HNIS concurs, that it is 
not possible, based on the information available, to establish 
the presence or absence of nonresponse bias. However, the 
likelihood of such bias cannot be disregarded. Nor is it 
possible to determine objectively the extent to which nonresponse 
bias might influence interpretation of analyses using these data. 


The Expert Panel concluded that between-group comparisons are 
possible but must be made with the recognition that the 
respondents may not be completely representative of the 
subgroups. The Panel also concluded that use of the data for 
estimates of specific foods or food groups, estimates of upper 
percentiles of intake, or estimates of intakes of subgroups for 
which the cell size is small is particularly questionable. 


HNIS suggests that users of these data carefully balance their 
need and the tolerance for error in their specific application 
against the potential for nonresponse bias in the 1987-88 NFCS 
dataset. Whenever possible, confirmatory data from other sources 
should be sought to support estimates based on analysis of these 
data. 


E. Nonresponse Report 


HNIS is preparing a report of the nonresponse investigations 
conducted by HNIS staff. It will include the report of the 
Expert Panel. Please write to us if you wish to receive a copy 
of the report when it is published. 
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The following are GAO’s comments on the Department of Agriculture’s 
letter dated May 22, 1991. 


1. HNIS’ action in October 1990 to draw attention to the survey’s low 


GAO’s Comments 


response rate in released documents was taken after GAO’s recommenda- 
tion to the Assistant Secretary in September 1990. As we mentioned in 
the report, we considered HNIS’ proposed documents to be insufficient, 
which is why we wrote the Assistant Secretary raising these concerns 
and recommending disclosure. See p. 23. 


2. In our opinion, the statement that HNIS developed in May 1991 over- 
states the response rate for households that provided individual intake 
data. The response rate should be based on the number of households 
that provided individual intake information, not all households that par- 
ticipated in the survey’s household food consumption component. Using 
the former reduces the household response rate from 38 to 34 percent. 
Before it is provided to data users, the statement USDA proposes should 
also include stronger cautionary language regarding use of the data 
because of their bias. 


3. We deleted this information from the report, since a technical discus- 
sion of the basis for the chosen methodology was unnecessary to 
describe the survey’s design. USDA’s comments on the methodological 
necessity of multiple-day intake data confirms that higher response 
rates are important; only one-quarter of the respondents in the NFCS’ 
basic sample provided more than 1 day of intake data. 


4. We revised the report to state that some participants who answered 
positively that they used all of the main food groups (in the 1987-88 
survey, there were 21) would have had to listen to the interviewer read 
a list of up to 398 food subgroups within those major groups. Contrary 
to USDA’s assertion, we are not advising HNIS how to reduce the burden 
on respondents. We raised this one option—reducing the number of food 
items to those with nutritional differences—in the 1977 report. 


5. We revised the report to include information provided in USDA’s com- 
ment, specifically, documentation of the 1977-78 sampling plan and indi- 
vidual intake response rate not previously provided to us. To our 
knowledge however, this description of the individual response rate was 
not reflected in the contractor’s 1980 report or in any other reports on 
the 1977-78 survey. 
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6. As stated on pp. 12 and 15 of the report, we agree that the 1987-88 
NFCS was very similar to the 1977-78 NFCS. 


7. We revised the report to state that the caPI method was introduced 
primarily to shorten the time needed for data processing. 


8. We revised the report to state that the weighting scheme adjusted for 
months of the year and several demographic characteristics. Under the 
adjustment for demographic characteristics, each individual received a 
weight reflecting that individual’s contribution relative to the 1987 U.S. 
population, with the sum of the weights equal to the total population. At 
least one individual represented as few as 1,000 persons and as many as 
136,000 persons. This disparity decreased the precision of any estimate 
that could be derived from the 1987-88 NFCs. 


9. We revised the report on p. 24 to state that HNIS is committed to exam- 
ining the impact of the low response rate on the low-income data. 


10. We revised the report to reflect deficient planning by HNIS’ program 
office in writing the contract in order to allow enough time for the con- 
tractor and HNIS to accomplish various tasks. 


11. We revised the report to state that the cost increases in the NFCS 
contract occurred between August 1988 and June 1989. The report does 
not ignore the root causes of the cost overruns. To assert that there was 
a government requirement to complete the survey, citing the ‘desire of 
Congress to receive the information” raises an unanswered question: Did 
USDA notify the Congress of the survey’s severe response rate problem 
(clearly evident in 1987), which meant that the survey’s data would be 
questionable at best and unuseable at worst? To our knowledge, USDA did 
not raise this issue with the Congress at the earliest time possible, when 
the Congress could have decided whether to commit additional govern- 
ment funds on a questionable survey. USDA should have informed the 
Congress about the problems with the survey. Furthermore, we recog- 
nize that part of the cause was that the survey and analysis required 
more work than anticipated. However, the agency’s and the contractor’s 
mismanagement during the administration phase (such as unauthorized 
expansion of data collection efforts) were key reasons why the survey 
involved more money than anticipated. 


12. We believe that USDA’s reasons for dropping the dress rehearsal do 


not adequately consider the drawbacks of not testing the contractor’s 
readiness to perform all aspects of the survey. Without a dress 
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rehearsal, USDA had no proof, for example, that National Analysts’ 


development of the CAPI method, including preparing interviewers to 
successfully use the method, was adequate. The problems that were 
reported with the interviewers’ performance using CAPI could have been 
avoided had a thorough dress rehearsal been conducted and evaluated. 


13. The report states that the contractor’s poor inventory control was 
one cause of the loss of the government’s lap-top computers. However, 
we disagree with USDA that the contract contained adequate instructions 
to control the return of government property. Under the NFCS contract, 
National Analysts was scheduled to finish using the computers by April 
1988, or 1 year before the contract was originally expected to be com- 
pleted. Subsequent contract modifications extended the contract until 
April 1991. Under uspA’s normal practice, National Analysts would have 
kept the lap-tops 3 years after they were finished using them. We ques- 
tion the lack of internal controls under the normal practice of leaving 
valuable government property with a contractor until whatever time the 
contract is completed, rather than until the task is finished. 


The lack of instructions for a more timely return of the property, com- 
bined with uUspaA’s and the contractor’s lax attention to property control 
procedures during the contract’s administration phase, delayed action 
on retrieving the computers (and financially recovering for the lost com- 
puters) until late 1990. By that time, the current market value of the 
unreturned lap-tops was much lower than the amount the government 
would likely have received had action been taken in 1988. It is also pos- 
sible that more lap-tops would have been returned to the government, 
since some may have been lost between 1988 and 1990. 


14. We generally agree with USDA’s comments on the contracting officer’s 
responsibilities and the need to rely on the technical knowledge and 
skills of the program staff, particularly the contracting officer’s repre- 
sentative. However, this does not absolve the contracting officer of 
responsibility for ensuring things are done right. Because of a con- 
tracting officer’s limited technical knowledge, it is imperative for the 
contracting officer to communicate substantively with technical repre- 
sentatives about a contractor’s progress and performance. Our report 
describes inadequate monitoring by FNS’ contracting officer and his staff 
between 1986 and 1988. During this time, when many problems were 
occurring with the contract, there was essentially no substantive com- 
munication between contracting and program office officials on the con- 
tract’s progress and status. 
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15. In an August 1990 meeting between three GAO representatives and 
FNS’ contract management branch chief and two of his staff, we dis- 
cussed our findings that the contracting office inadequately monitored 
the NFCS contract during a critical period from 1986 well into 1988. Our 
findings were based largely on direct discussions with the two contract 
specialists who served for the first 28 months of the contract and on a 
review of the official contract file. The first contract specialist, who 
served for 17 months, told us she did not deal with the cor during her 
tenure. She further stated that, as a contract specialist, she was to place 
all documentation, including records of phone conversations, concerning 
the contract into the official contract file. The second contract specialist 
told us that, except for instructions on processing invoices, she had no 
regular discussions with the cor until July 1988, when National Ana- 
lysts notified her office that contract funds were exhausted and almost 
$1 million was needed to finish the work. Even so, she reported great 
difficulty in obtaining necessary information from the Cor in order to 
negotiate the contract’s subsequent modifications. Their accounts were 
corroborated by our review of the contract file, which contained no doc- 
umentation of any substantive communication about problems with the 
contract’s progress or status between the contracting office, HNIS, or the 
contractor. 


The contract management branch chief did not refute our findings 
during the August 1990 meeting. Furthermore, he said that when he 
came to the branch in 1988, he recognized that basically no contract 
administration was going on with the 1987-88 NFCS contract at either FNS 
or HNIS. Our report discusses efforts taken by the branch chief after 
1988 to improve the NFCS contract’s administration, including converting 
to a fixed-price contract in 1989 to stem further cost overruns. Gener- 
ally speaking, after 1988, the contracting office’s corrective actions 
were too late to reverse the damage done. 
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report text appear at the 


end of this appendix. NATIONAL ANALYSTS 


DIVISION OF BOOZ*ALLEN & HAMILTON INC. 
1700 MARKET STREET « PHILADELPHIA, PENNSYLVANIA 19103 » TELEPHONE: (215) 496-6800 » FAX: (215) 496-6801 


MARSHALL G. GREENBERG, Ph.D. 
President 


May 21, 1991 


Mr. John W. Harman 
Director, Food and 
Agriculture Issues 
U.S. General Accounting Office 
Resources, Community and 
Economic Development Division 
Washington, DC 20548 


Dear Mr. Harman: 


This is in response to your letter of April 15, 1991 regarding a proposed draft of a GAO 
report (GAO/RCED-91-117) inviting written comments. I have read the excerpt of the 
pre osed report and take exception to the title which is misleading in light of the facts. 

the remainder of this letter, I have attempted to shed light on some of the report's 
inaccuracies on a page-by-page basis. 


Generally, the data from the NFCS study are now under review and meaningful 
See comment 1. information will be available to many government agencies and other users of these 
data. Clearly, there were problems using a new data collection method in as 
Sate and complex a survey research effort as the Nationwide Food Consumption 
Survey. However, National Analysts attempted to address problems as they arose and 

erformed the requirements of its contract. I trust any final report issued by GAO will 
include the information we are providing. 


* * * * * 
Now on p. 17. Page 5 -- The statement that "HNIS compounded the flaws in the design of the NFCS by 
rematurely introducing a major methodological change from previous surveys" 
See comment 2. implies that there are numerous and significant design problems with the NFCS in 


addition to computer-aided data collection. This statement is not supported either in 
the arguments presented in the report, nor by independent review of the NFCS and 
other surveys using parts of the same methodology that was employed by the NFCS 
project. 


The general design is the same as that employed in NFCS 77/78 and all the 
supplementary surveys previously cited in Chapter I of the report. Methodological 
studies prior to both the NFCS 77/78 and the NFCS 87/88 were conducted by USDA to 
refine and improve the design. Inter-agency and external reviews of the approach were 
conducted and judged the design to be sound. National Analysts was not aware of any 
design "flaws" when the research effort was begun. 


ET a eee 
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Mr. John W. Harman 
May 21, 1991 
Page 2 


Now on p. 17. The statement that, "These problems might have been avoided had the CAPI method 
been better developed by HNIS and National Analysts" suggests that a computer-aided 
interview program was deficient, a claim which is not substantiated. At the time the 
NECS research was requested, a review of the relevant literature showed there was very 
little published experience available to guide the development of a CAPI interview. 
National Analysts had its own experience, based on research for USDA to draw upon. 
Most of the problems identified in that earlier effort (e.g., weight of the "luggable" 
computer, ce printing time for questions to appear) were ones taken into account 
when the CAPI program was designed. Given the nature and complexity of the survey 
data to be captured, the program is generally thought to work very well. Presentation 
of the concept and demonstrations of the interview in action in public forums (e.g., 
Bureau of the Census Fourth Annual Research Conference -- 1988) has brought positive 
comments. 


See comment 3. 


Information regarding the pretest by another contractor was not made available to 
National Analysts in a timely manner. The data quality problems Aer by the 
other contractor were thought not to be likely to recur because of the differences 
between their program and the ones employed in the NFCS effort. 


National Analysts is in agreement with the statement that some of the problems with 
the CAPI might have been averted if there had been time to conduct more testing of the 
NFCS programmed questionnaire and to test the post-field procedures. However the 
window for extensive pretesting of the CAPI was small according to the original 
contract schedule and continued to shrink due to factors beyond our control. The set of 
individual questionnaire items continued to change through late November 1986, thus 
eroding time available to National Analysts for testing. 


Critical data required to program the CAPI were not provided by HNIS until well after 
the time those data were needed to conduct a dress rehearsal. The food use data were 
the most important elements in the CAPI questionnaire. In order to program the 
aut contingency questions about the food items respondents reported using, 

IS needed to identify for National Analysts programmers extensive information 
about the item that existed in the HNIS database len the form, variation and weight 
information for the item, a "linkage" code needed to tie the interview data to the 
nutrient data base). National Analysts noted in its progress reports as late as mid- 
February 1987, which was many weeks after the data should have been supplied, that 
"Section II (the food use data) programming continues to be delayed because of the 
slow delivery of the weight book and NA/USDA linkage information" from HNIS. 


Changes in the content of the questionnaire continued to be introduced by USDA and 
OMB up to three weeks prior to the full- scale data collection effort and less than two 
weeks prior to the date materials needed to be distributed to the planned interviewer 
training sessions. Content changes requiring programming work were communicated 
to National Analysts on February 27, March 2, and 6, 1987. Given the timing and 
frequency of these changes, there was little time to field test the actual use of the CAPI 
or of the post-field procedures. Delaying the onset of the national study was not 
considered an option. National Analysts, as the contractor for this assignment, did not 
have the latitude to change the start of the data collection period. 


Now on p. 18. Page 6 -- The statement that HNIS and National Analysts canceled the dress 
rehearsal..."after the contractor had fallen behind schedule" suggests that National 


See comment 4. 
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Analysts caused these slippages. This is not a correct interpretation of the situation, for 
all the reasons stated above. National Analysts reacted promptly to all the input and 
changes in the questionnaire content which USDA introduced. Moreover, whether or 
not to cancel the dress rehearsal was not the contractor's decision to make. That was the 
call of the HNIS. 


Now on p. 18. National Analysts disagrees with the statement that "The contractor changed the 
survey's design in ways that may have undermined the data quality." The changes 
introduced were not fed changes but procedural changes implemented to deal with 
problems as they emerged in the field and coding areas. The changes were introduced 
to combat difficulties and to sustain quality, not to undermine it. 


See comment 5. 


Now on pp. 18-20. Page 7 — National Analysts agrees with most of the explanation of completion rates 
offered in the report in this senior 


Interviewers repeatedly reported in their regular telephone contacts with the National 
Analysts’ central office field supervisors that the mere perception of the interview 
burden was discouraging to most respondents. Whether or not the respondents knew 
the exact length of the interview (which varied from house to house), the fact that they 
were being asked to do recordkeeping for a week before the interview and to schedule 
several hours for the interview appointment caused many, in the minds of interviewers, 
to refuse participation. 


The lack of meaningful monetary compensation for respondents’ time was the other 
major problem voiced by interviewers to their National aes field supervisors. It is 
known from the other studies, including the Continuing Survey of Food Intake by 
Individuals, currently being conducted by National Analysts for HNIS and which has a 
significantly higher completion rate, that interviewers find the $2.00 offer for the 
completion of the intake records to be of little help in gaining cooperation. In the CSFII, 
the perceived length of time for the interview is the greatest barrier to participation — 
ena the CSFII interview is much shorter than the NFCS) even among those se could 
not be reached at the time of interview. HNIS made it clear to National Analysts that 
offering more of a financial incentive to all participants was not possible. 


Significant efforts were made to improve the completion rate. These included such 
activities as: 


e Sending letters and making telephone calls to potential respondents from the 
Philadelphia office 
e Providing escorts to interviewers working in difficult areas 


e¢ Reassigning sample households to different interviewers for follow-up contact 
and attempted interviews 


e Offering interviewers incentives for timely accurate performance 


e Sending letters to managers of locked buildings to solicit opportunities to gain 
entrance 
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e Traveling expert interviewers into areas 


e Seeking local publicity to promote visibility for the survey and increase interest 


Now on pp. 20-21. Page 9 -- The NFCS interviewing began very slowly and oe were immediately taken 
to rectify this. National Analysts exerted its best efforts, both to complete as many 
interviews as possible, and to spread these interviews equally throughout the data 
collection period to the extent possible. The potential to extend the data collection was 
an idea first suggested to National Analysts by members of the HNIS site team in a 
meeting in early September, 1987. National Analysts was anxious to gather as many 
interviews as possible and, therefore, when it became clear in the Winter quarter that 
the targeted number of interviews would not be completed by the end of March, it was 
suggested that the fieldwork period be extended. National Analysts believed this was 
consistent with HNIS' goals as stated to us earlier, and could help the overall 
performance even if it would delay the date for completion. 


Now on pp. 20-21. The distribution of interviews across the days of the week was approached on a best 
S t6 efforts basis as noted in National Analysts’ initial proposal. Sunday interviewing is 

spre seals} traditionally less frequent because respondents are less willing to participate in surveys 
in some areas on Sundays. In its Request for Proposal, USDA recognized this limitation 
and specified that "not less than 10% of the total completed household food 
consumption eo es (be taken) during each day — with the possible exception 
of Sunday." e results of the survey by day of the week are consistent with this 
Sag neti The distribution of interviews from Monday through Saturday shows that 
each captured more than the 10% minimum that was specified. 


Now on p. 22. Page 10 -- The discussion of the issue of interviewer selection and training is 
misleading. The goal had been to have approximately 200 interviewers working on the 
See comment 7. project at any given time. National Analysts, however, did anticipate that in a study of 


this length that there would be turnover. In 12 months, some interviewers would get 
sick, others would become tired of the study, others would be released in favor of better 
interviewers. This was expected and planned for. We believe that the computer 
interviewing, coupled with the demanding nature of the data collection, caused more 
than the typical amount of turnover to occur. Turnover was about 20% greater than 
expected, not 50% as the report suggests. 


Now on pp. 22-23. Page 11 -- The formal training sessions for interviewers, which lasted 5 days, were 
designed for large groups of interviewers of between 20 and 25 people. Replacement 
training was done in smaller groups of 1 to 3 interviewers allowing for considerably 
more personal attention. All of the same material was covered ie the replacement 
interviewers in these sessions as was addressed in the larger groups. Interviewers in all 
sessions were trained both in the use of the computer and in the paper-and-pencil 
questionnaire as the latter was always intended as a yaiciop! 


See comment 8. 


Now on pp. 22-23. The draft report states, "Quality controls were also relaxed for the data processing staff, 
pce coders, possibly jeopardizing the quality of the data developed for the 

FCS survey." This is not correct. There were numerous sets of checks and edits built 
into the data processing system from double coding or data entered by junior coders to 
computerized logic and reasonableness checks. Our initial plans called for five-day 
training sessions with brand new coders. These types of sessions were tried and found 


See comment 9. 
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not to be effective in bringing coders up to standards. The complexity of the coding and 
editing responsibility, coupled with the changing nature of the task (the USDA coding 
system is Ren and requires constant updating to reflect the current changes in 
foods available in the marketplace), proved to be beyond the abilities of many clerical 
personnel. Many clerks resigned from the project during, or shortly after, the formal 
training sessions. Others were terminated due to poor performance. 


Because of high turnover in the early stages of the project, much of National Analysts’ 
coding management time was spent in recruiting and conducting training sessions 
which were determined to be unproductive. The program was, therefore, changed -- 
selectively using temporary services to screen candidates according to National 
Analysts’ specifications. National Analysts' personnel conducted all the training and 
maintained the supervision of these clerical personnel, although they remained on the 
ee roll of the temporary service (a fact that was duly noted in our monthly progress 


ngs to USDA). 
Now on pp. 22-23. Fage 12 — The training of coders was changed in order to improve the retention rate 
Ss ne euler and to improve the quality of their work. Following an abridged training briefing that 


was adapted to make the coding requirements less difficult to understand, new coders 
worked directly with more experienced individuals who continued their training, 
reviewed and corrected their errors and gave immediate feedback. This individualized 
approach was judged by coding management to be more effective than the initial five- 


ay training. 
Now on p. 31. Page 15 -- The comparison in the footnote between the price paid FNS for the missing 


CoatecRnentalt: used laptop oes in 1990 and the price paid originally in 1986, is definitely 


misleading. It fails to recognize the dramatic es and evolution during this time in 
the laptop market, which led to devaluation of all used computer equipment. The $300 
er unit paid to FNS is the amount specified by the government contracts office in its 
etter to National Analysts of ota 6, 1990. Two vendors of used computer 
equipment contacted by National Analysts would not even offer a price for the laptops. 


I trust this information clarifies some of the issues raised in your report, and that the 
misstatements will be rectified. 


Sincerely yours, 


Marshall G. Greenberg NATIONAL ANALYSYS 
President A Division of Booze Allen 
& Hamilton Inc. 


MGG/hr 
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The following are GAO’s comments on National Analysts’ letter dated 
Mayicie 991° 


1. Contrary to National Analysts’ views, the review of the NFCS data by 
an independent expert panel confirmed our findings that the survey has 
serious limitations largely due to low response rates. 


Furthermore, we disagree that National Analysts performed the require- 
ments of its contract with USDA. As we document in this report, National 
Analysts did not perform all the contract’s requirements (such as the 
dress rehearsal); nor did the contractor deliver any data from a required 
follow-up surveys of nonrespondents. The omission of the dress 
rehearsal is particularly troublesome. According to National Analysts’ 
officer-in-charge and HNIS’ former contracting officer’s representative, 
they jointly decided to forego the dress rehearsal. They did this without 
consulting FNS’ contracting officer, who, under the terms of the contract, 
was the only official with authority to waive requirements. 


2 Our chief criticism of the survey is its burden on respondents, a meth- 
odological flaw we first identified in our 1977 report on the 1977-78 
nrcs. National Analysts clearly agrees with this assessment on p. 3 of its 
letter, which describes in detail the survey’s burden on respondents. The 
expert panel also found that the NFrcs’ burden on respondents was very 
great and that changes in the survey’s design to alleviate this might 
improve response rates in the future. 


3. We revised the report to state that the problems with cap! might have 
been avoided had HNIS and National Analysts better developed and fully 
tested the method before implementation. The report does not criticize 
the concept of the caPI method for collecting NFcs data. The reasons we 
report inadequate development of the method are consistent with the 
account National Analysts gives. 


4. In chapters 2 and 3, we discuss UsDa’s contribution to the ““slippages”’ 
that led to the cancellation of the dress rehearsal. According to National 
Analysts’ officer-in-charge and HNIs’ former CoR, it was a joint decision 
to cancel the dress rehearsal. The issue was never raised with FNS con- 
tracting officer. The contracting office is the only official the contract 
authorized to waive or modify the contract’s requirements. Conse- 
quently, even if HNIS alone directed National Analysts to forgo the dress 
rehearsal, National Analysts was not relieved from the contractual 
requirement to hold a dress rehearsal. 
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5. We revised the report to state that the contractor deviated from pro- 
cedures specified in the survey’s design in ways that may have under- 
mined the data’s quality. As we document in the remainder of chapter a, 
these deviations (such as reducing quality controls over the interviewers 
and data processors and not following prescribed schedules) probably 
lowered the data’s quality. 


6. We deleted information in the report on the methodology for sampling 
households over the days of the week. 


7. We revised the report to reflect only that the turnover of interviewers 
was greater than expected. 


8. The evidence in our report substantiates that National Analysts’ 
training of interviewers was inadequate. For example, both National 
Analysts and HNIS reported that interviewers had many difficulties 
using the computer method. Reducing the time for training the inter- 
viewers may have contributed to this problem. Furthermore, if National 
Analysts had expected to train small groups of replacement inter- 
viewers, its technical proposal should have stated that it planned to 
shorten the training. It did not do so, and thus, the contract required a 
minimum 5-day training session for all interviewers. 


9. Given National Analysts’ past experience conducting the NFCS and 
Continuing Surveys for HNIS, we question why the company did not 
know that the rigorous training and monitoring of data coders would be 
difficult to implement. Instead, National Analysts included rigorous 
training and monitoring of these staff in its technical proposal, and the 
training plan was incorporated as one of the contract’s requirements, 
but National Analysts quickly abandoned the plan once the project was 
under way. This not only violated the contract’s requirements, but also 
changed procedures in a way that may have undermined the quality of 
the NFcs data. According to HNIs officials, data processing was poor in 
quality and unreasonably delayed. 


10. We do not disagree that the laptops’ value depreciated from 1987 to 
1990. If the reimbursement had occurred in a more timely manner (that 
is, when the units were lost, stolen, or damaged between 1987 and 
1988), it is likely that the current market value would have been closer 
to the laptops’ original value. 
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Response Rate for 
Households 


Table IIl.1: Response Rates for 
Households Providing Individual Intake 
Data 


Estimated Response 
Rate for Individuals 


Table III.2: Estimated Response Rate for 
Individuals Providing First Day of Intake 
Data? 


The response rate for households for the individual intake component of 
the 1987-88 NECS was 33.8% (see table III.1). Thus, for every 100 house- 
holds expected to participate, fewer than 34 contributed some individual 
intake information. This rate may differ somewhat once the contractor 
submits final data for household food use, but should still be low. Our 
analysis of preliminary tabulations from National Analysts indicates 
that the response rate for households in the low-income sample was 
about 45 percent. 


SETTLE 


Number of households contractor attempted to reach 13,733 
Number of households deemed vacant ies ey 
Number of households with occupant(s) 12,181 

Number of responding households? 4,114 

Response rate for households (4,114/12,181) 33.8% 


alndividual intake information provided for at least one person. 
Source: HNIS. 


As can be expected with any survey, not all individuals within cooper- 
ating households participated. The estimated response rate for individ- 
uals for the first day of intake was 31.1 percent (see table III.2) Thus, 
for every 100 individuals, fewer than 32 offered food intake information 
for the first day. Not all individuals who participated the first day chose 
to provide all 3 days of intake data (see table III.3). 


LL 


Number of individuals providing data 10,172 
Number of individuals in responding households 11,045 
Response rate for households (see table Ill. 1) 33.8% 
Percentage of participating individuals within responding households 

(10,172/11,045) 92.1% 
Response rate for individuals providing first day of intake data (92.1% X 

33.8%) 31.1% 


Sennen eee. 
a\We do not know how many individuals were in the 13,733 households targeted for the sample. Thus, to 
estimate response rates for individuals, we assumed that the average sizes of both nonresponding and 
responding households were the same. 


Source: HNIS. 
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Table III.3: Estimated Response Rates 
for Individuals Providing Intake Data for 
3 Days? 


Number of individuals providing all 3 days of intake data 8,468 
Number of individuals providing first day of intake data 10,172 
Number of individuals in all households 11,045 
Response rate for households (see table Ill. 1) 33.8% 
Percentage of individuals within responding households providing first day of 

intake data (10,172/11,045) (see table Ill.2) 92.1% 
Percentage of individuals providing first day of intake data who later 

provided data for all 3 days (8,468/10,172) 83.2% 
Response rate for individuals providing 3 days of intake data 

(83.2% X 92.1% X 33.8%) 25.9% 


“We do not know how many individuals were in the 13,733 households targeted for the sample. Thus, to 
estimate response rates for individuals, we assumed that the average sizes of both nonresponding and 
responding households were the same. 


Source: HNIS. 
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